如何提高过滤系统中 Laravel 多对多关系的查询速度？

Question

我用 laravel 8 建立了一个网站。服务器是 6 核 CPU / 6 GB Ram VPS。服务器是 Linux CentOS with nginx 和 mysql 8.

高峰期同时在线访问量在500左右。 CPU高峰期100%，其余时间>80%。

我检查了使用情况，发现大部分资源都被 mysql 使用了。然后我找到了一些缓慢的查询，我认为这个 many-to-many 关系查询是主要原因之一。

存在具有 many-to-many 关系设置的视频模型和流派模型。在视频table中，大约有800,000行。 genre table 中有 700 多个流派，genre_video table 中有 237,4344 种关系。 videos.id和genres.id分别是videos和genrestable的主索引。外键设置在genre_video

视频模型

class Video extends Model
{
   use HasFactory;
   public function genres()
    {
        return $this->belongsToMany(Genre::class);
    }

}

类型模型

class Genre extends Model
{
    use HasFactory;
    public $timestamps = false;
    public function videos()
    {
        return $this->belongsToMany(Video::class);
    }
}

表格

videos
id           video_info1           video_info2          type_code
1            somethining           somethining          1
2            somethining           somethining          1
3            somethining           somethining          1

genres
id           genre_name
1            G1
2            G2
3            G3
4            G4
5            G5


genre_video
genre_id         video_id
1                1
1                3
1                5
2                1
2                3


previews (one-to-one with video)
id          image
1           aaa.jpg
2           bbb.jpg
3           ccc.jpg


titles (one-to-one with video)
id          title
1           aaa
2           bbb
3           ccc

过滤函数

我的网站上有一个流派列表。当访问者点击流派时，它会更改 url。

例如： Gerne 列表：G1 G2 G3 G4 G5

当访问者点击 G1 时，url 变为 /?c=1

然后访客点击G3，url变成/?c=1,3

然后访客点击G5，url变成/?c=1,3,5

该函数将获取所有选定的流派 ID 作为数组 $cArr。然后我使用 whereHas 循环遍历数组以查找与类型 1、3、5 匹配的所有视频。随着访问者在过滤器中添加更多类型，他们可以准确地找到他们想要的。即示例中 id = 1 的视频。但是这个查询大约用了 20-50 秒。

if($request->c){
                $c = $request->c;
                $cArr = explode(',',$c);
                $data = Video::where('type_code',$type_code)
                            ->whereHas('genres',function ($query) use($cArr) {
                                $query->whereIn('genres.genre_id', $cArr);
                                }, '=', count($cArr))
                            ->join('previews','previews.code','=','videos.code')
                            ->join('titles','titles.code','=','videos.code')
                            ->orderBy('publish_date', 'DESC')
                            ->limit(400)->get();
}

$type_code 只会等于 0,1,2,3
预览和标题与视频有one-to-one关系

我的问题是：

有没有办法让这个查询更好，同时保持过滤功能？
我在网上查过有人说我们应该使用 sphinex 这样的索引引擎。我不知道它是否与我的设置兼容linux + centOS 7 + nginx + mysql 8 + laravel 8。关于使用索引引擎有什么建议吗？

更新

感谢您花时间阅读我的问题。以下是实际生成的查询的一些示例。时间已经是最佳速度，因为它在一天中的流量最少。

第一个是访问者点击 G2 和 G13 时。他将看到 400 个类型为 G2 和 G13 的视频。

select * from `videos` 
inner join `previews` on `previews`.`code` = `videos`.`code` 
inner join `titles` on `titles`.`code` = `videos`.`code` 
where `type_code` = 0 and (
    select count(*) 
    from `genres` 
    inner join `genre_video` on `genres`.`id` = `genre_video`.`genre_id` 
    where `videos`.`id` = `genre_video`.`video_id` 
    and `genres`.`genre_id` in ('2', '13')
) = 2 order by `publish_date` desc limit 400

Query took 13.58s

第二个是访问者点击 G2、G13 和 G18 时。他甚至会看到所有这些流派的 400 个经过精确过滤的视频

select * from `videos` 
inner join `previews` on `previews`.`code` = `videos`.`code` 
inner join `titles` on `titles`.`code` = `videos`.`code` 
where `type_code` = 0 and (
    select count(*) from `genres` 
    inner join `genre_video` on `genres`.`id` = `genre_video`.`genre_id` 
    where `videos`.`id` = `genre_video`.`video_id` 
    and `genres`.`genre_id` in ('2', '13', '18')
) = 3 order by `publish_date` desc limit 400

Query took 14.04s

更新 2

我添加了列、索引和关系屏幕截图。很抱歉，我无法提供 laravel 迁移文件，因为我在学习迁移之前在 phpmyadmin 中创建了这些 table。但似乎所有必需的关系都是根据 many-to-many 文档添加的。

actors

videos

actor_vidio

previews

titles

再次抱歉让问题变得混乱。

更新 3

EXPLAIN select * from `videos` 
inner join `previews` on `previews`.`code` = `videos`.`code` 
inner join `titles` on `titles`.`code` = `videos`.`code` 
where `type_code` = 0 and (
    select count(*) from `genres` 
    inner join `genre_video` on `genres`.`id` = `genre_video`.`genre_id` 
    where `videos`.`id` = `genre_video`.`video_id` 
    and `genres`.`genre_id` in ('2', '13', '18')
) = 3 order by `publish_date` desc limit 400

EXPLAIN select * from `videos` 
inner join `previews` on `previews`.`code` = `videos`.`code` 
inner join `titles` on `titles`.`code` = `videos`.`code` 
where `type_code` = 0 and (
    select count(*) from `genres` 
    inner join `genre_video` on `genres`.`id` = `genre_video`.`genre_id` 
    where `videos`.`id` = `genre_video`.`video_id` 
    and `genres`.`genre_id` in ('2', '13', '18')
) = 3 order by `publish_date` desc limit 400

更新 4

我加了SHOW CREATE TABLE

actors

actors
CREATE TABLE `actors` (
 `id` int unsigned NOT NULL AUTO_INCREMENT,
 `actor_id` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs NOT NULL,
 `actor_type` int unsigned NOT NULL DEFAULT '0',
 `actor_img` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `actor_sex` int DEFAULT '2',
 `actor_cn` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `actor_tw` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `actor_en` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `actor_ja` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `actor_ko` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `created_at` timestamp NULL DEFAULT NULL,
 `updated_at` timestamp NULL DEFAULT NULL,
 PRIMARY KEY (`id`),
 UNIQUE KEY `actor_id` (`actor_id`),
 KEY `actor_sex` (`actor_sex`)
) ENGINE=InnoDB AUTO_INCREMENT=89588 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

videos

CREATE TABLE `videos` (
 `id` int unsigned NOT NULL AUTO_INCREMENT,
 `type_code` int NOT NULL,
 `code` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
 `publish_date` date NOT NULL,
 `duration` int NOT NULL,
 `download` int NOT NULL,
 `sub` int NOT NULL,
 `online` int NOT NULL DEFAULT '0',
 `leak` int NOT NULL DEFAULT '0',
 `javdb_url_code` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
 `created_at` timestamp NULL DEFAULT NULL,
 `updated_at` timestamp NULL DEFAULT NULL,
 `is_single_actor` int NOT NULL DEFAULT '0',
 PRIMARY KEY (`id`),
 UNIQUE KEY `code` (`code`),
 KEY `type_code` (`type_code`),
 FULLTEXT KEY `code_fulltext` (`code`)
) ENGINE=InnoDB AUTO_INCREMENT=458527 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

actor_video

CREATE TABLE `actor_video` (
 `video_id` int unsigned NOT NULL,
 `actor_id` int unsigned NOT NULL,
 PRIMARY KEY (`video_id`,`actor_id`),
 KEY `actor_video_actor_id_foreign` (`actor_id`) USING BTREE,
 KEY `actor_video_video_id_foreign` (`video_id`) USING BTREE,
 KEY `actor_id` (`actor_id`),
 KEY `video_id` (`video_id`),
 CONSTRAINT `actress_video_actress_id_foreign` FOREIGN KEY (`actor_id`) REFERENCES `actors` (`id`) ON DELETE CASCADE ON UPDATE RESTRICT,
 CONSTRAINT `actress_video_video_id_foreign` FOREIGN KEY (`video_id`) REFERENCES `videos` (`id`) ON DELETE CASCADE ON UPDATE RESTRICT
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

previews

CREATE TABLE `previews` (
 `id` int unsigned NOT NULL AUTO_INCREMENT,
 `code` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `video_preview` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `image_pl` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `image_ps` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `image_preview_s` varchar(3000) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `image_preview` varchar(3000) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 PRIMARY KEY (`id`),
 KEY `code` (`code`)
) ENGINE=InnoDB AUTO_INCREMENT=458096 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

titles

CREATE TABLE `titles` (
 `id` int unsigned NOT NULL AUTO_INCREMENT,
 `code` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
 `title_cn` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `title_tw` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `title_en` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `title_ja` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `title_ko` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 PRIMARY KEY (`id`),
 KEY `code` (`code`),
 FULLTEXT KEY `title_index` (`title_ja`,`title_en`) /*!50100 WITH PARSER `ngram` */ ,
 CONSTRAINT `title_video_fk` FOREIGN KEY (`code`) REFERENCES `videos` (`code`) ON DELETE CASCADE ON UPDATE RESTRICT
) ENGINE=InnoDB AUTO_INCREMENT=458101 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

genres

CREATE TABLE `genres` (
 `id` int unsigned NOT NULL AUTO_INCREMENT,
 `genre_id` int NOT NULL,
 `genre_type` int NOT NULL DEFAULT '0',
 `genre_cn` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `genre_tw` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `genre_en` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `genre_ja` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `genre_ko` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `type_code` int NOT NULL,
 PRIMARY KEY (`id`),
 UNIQUE KEY `genre_id` (`genre_id`),
 FULLTEXT KEY `genre_cn` (`genre_cn`,`genre_tw`,`genre_en`,`genre_ja`)
) ENGINE=InnoDB AUTO_INCREMENT=1536 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

genre_video

CREATE TABLE `genre_video` (
 `genre_id` int unsigned NOT NULL,
 `video_id` int unsigned NOT NULL,
 KEY `genre_video_genre_id_foreign` (`genre_id`),
 KEY `genre_video_video_id_foreign` (`video_id`),
 KEY `genre_id` (`genre_id`),
 CONSTRAINT `genre_video_genre_id_foreign` FOREIGN KEY (`genre_id`) REFERENCES `genres` (`id`) ON DELETE CASCADE,
 CONSTRAINT `genre_video_video_id_foreign` FOREIGN KEY (`video_id`) REFERENCES `videos` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

Answer 1

Many:to:many 表的索引往往很差，导致很多额外的 CPU。这显示了最佳架构（无 auto_inc）和索引（2 个复合索引）：

http://mysql.rjweb.org/doc.php/index_cookbook_mysql#many_to_many_mapping_table

Answer 2

首先，您的 JOIN 公用键是 code。

在titles

中是这样定义的

code varchar(50)

并在 videos 和 previews

中像这样

code varchar(255)

这对 ON 条件下的性能不利。完全相同地定义所有三个 code 列。

下一期：告别依赖子查询

我们可以写

              select genre_video.video_id
                     from genres
                     join genre_video ON genres.id = genre_video.id
                     join videos on genre_video.video_id = videos.id
                    where genres.genre_id IN (2, 13, 18)
                      and videos.type_code = 0
                 group by genre_video.video_id
                   having COUNT(*) = 3

获取与三种流派匹配的 video_id 值。

然后我们将子查询加入另一个子查询。这会完成 order by ... limit 400.

的繁重工作

      select code
        from (
                   select genre_video.video_id
                     from genres
                     join genre_video ON genres.id = genre_video.id
                     join videos on genre_video.video_id = videos.id
                    where genres.genre_id IN (2, 13, 18)
                      and videos.type_code = 0
                 group by genre_video.video_id
                   having COUNT(*) = 3
             ) matches
        join videos ON matches.video_id = videos.id
       order by publish_date desc
       limit 400

然后我们才加入各种表并执行 select *

select *
  from (
          select code
            from (
                  select genre_video.video_id
                         from genres
                         join genre_video ON genres.id = genre_video.id
                         join videos on genre_video.video_id = videos.id
                        where genres.genre_id IN (2, 13, 18)
                          and videos.type_code = 0
                     group by genre_video.video_id
                       having COUNT(*) = 3
                 ) matches
            join videos ON matches.video_id = videos.id
           order by publish_date desc
           limit 400
       ) chosen
  join videos on chosen.code = videos.code
  join titles on chosen.code = titles.code
 order by videos.publish_date;

这应该很有帮助。

抱歉，我不知道如何在 Laravel 中编写此代码。

你没有给我们看genres，但它应该有这两个索引，这样的事情才能有效地工作。它不需要任何一个列上的索引。

PRIMARY KEY (video_id, genre_id)
INDEX (genre_id, video_id)

（顺便说一下，这条关于索引的建议也适用于 actor_video。）

像这样的棘手问题才是真正应用的标志。随着应用程序的增长，您应该监控性能。您可能需要其他索引，或者需要重构其他查询。

如何提高过滤系统中 Laravel 多对多关系的查询速度？

How to improve query speed on Laravel many to many relationship in a filter system?

php

mysql

many-to-many

query-optimization

laravel

更新

更新 2

更新 3

更新 4