如何更快地检索 500k 记录的数据库数据？

Question

我有两个表 T1 1 000 条记录和 T2 500 000 条记录。我有一个查询，其中我运行它们之间的连接并通过执行一些聚合来获取数据。我的页面似乎加载缓慢。有没有什么方法可以使这个查询更快？

我已经在执行聚合的列上创建了索引。我认为这是一个通用的声明。

      $query = Mymodel::selectRaw("supplier_data.name as distributor,supplier_data.name as name, supplier_data.group_id as group_id, supplier_data.pay,supplier_data.group_id as submitted_group_plan,supplier_data.group_id as group_id_string,
            (SELECT sum(t.net_claim) AS trans_number 
            FROM transactions_data_new as t 
            JOIN  `supplier_data` AS d ON  `t`.`member_id` =  `d`.`group_id`
            WHERE
            (
                (
                t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to' 
                AND t.`member_id` = supplier_data.group_id
                )
                OR
                (
                    (t.claim_status  IS NULL)
                    AND
                    (t.submit_date is NULL)
                )
            )
            AND d.id = supplier_data.id
        ) as trans_number,


        (SELECT sum(t.claim) AS trans_number 
            FROM transactions_data_new as t 
            JOIN  `supplier_data` AS d ON  `t`.`member_id` =  `d`.`group_id`
            WHERE
            (
                (
                t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to' 
                AND t.`member_id` = supplier_data.group_id
                )
                OR
                (
                    (t.claim_status  IS NULL)
                    AND
                    (t.submit_date is NULL)
                )
            )
            AND d.id = supplier_data.id
        ) as claim,

        (SELECT sum(t.reversed) AS trans_number 
            FROM transactions_data_new as t 
            JOIN  `supplier_data` AS d ON  `t`.`member_id` =  `d`.`group_id`
            WHERE
            (
                (
                t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to' 
                AND t.`member_id` = supplier_data.group_id
                )
                OR
                (
                    (t.claim_status  IS NULL)
                    AND
                    (t.submit_date is NULL)
                )
            )
            AND d.id = supplier_data.id
        ) as reversed,

        (SELECT sum(t.reversal) AS trans_number 
            FROM transactions_data_new as t 
            JOIN  `supplier_data` AS d ON  `t`.`member_id` =  `d`.`group_id`
            WHERE
            (
                (
                t.`submit_date`>= '$date_from' and t.`submit_date`<= '$date_to'
                AND t.`member_id` = supplier_data.group_id
                )
                OR
                (
                    (t.claim_status  IS NULL)
                    AND
                    (t.submit_date is NULL)
                )
            )
            AND d.id = supplier_data.id
        ) as reversal
            ");

Answer 1

我也没有看到这个需求 complex/repeated 相同的子句和多个子选择相同 table 可以使用单个左连接

SELECT 
  s.name AS distributor,
  s.name AS name,
  s.group_id AS group_id,
  s.pay,
  s.group_id AS submitted_group_plan,
  s.group_id AS group_id_string,
  SUM(t.net_claim) AS trans_number,
  SUM(t.claim) AS claim,
  SUM(t.reversed) reversed,
  SUM(t.reversal) reversal 
FROM
  supplier_data s 
  LEFT JOIN transactions_data_new t 
    ON `t`.`member_id` = s.`group_id` 
    AND (
      (
        t.`submit_date` >= '$date_from' 
        AND t.`submit_date` <= '$date_to'
      ) 
      OR (
        t.claim_status IS NULL 
        AND t.submit_date IS NULL
      )
    ) 
GROUP BY s.name,
  s.group_id,
  s.pay

Answer 2

据我了解，chunk() 方法适用于需要处理大型数据集并逐块对该数据执行操作的情况。

从你的问题来看，听起来你正在执行查询然后将数据返回为 JSON 所以对我来说，这听起来不像你正在对需要分块的数据集执行操作.

如果您想分解返回的 JSON 数据，您应该改为查看分页。

您可以像这样对查询应用分页：

$data = Inspector::latest('id')
    ->select('id', 'firstname', 'status', 'state', 'phone')
    ->where('firstname', 'LIKE', '%' . $searchtext . '%')
    ->paginate();

您可以通过将数字传递给分页方法来指定每个集合的大小：

$data = Inspector::latest('id')
    ->select('id', 'firstname', 'status', 'state', 'phone')
    ->where('firstname', 'LIKE', '%' . $searchtext . '%')
    ->paginate(25);

如果我误解了你确实想做分块，我相信你可以这样做：

$data = Inspector::latest('id')
    ->select('id', 'firstname', 'status', 'state', 'phone')
    ->where('firstname', 'LIKE', '%' . $searchtext . '%')
    ->chunk(50, function($inspectors) {
        foreach ($inspectors as $inspector) {
            // apply some action to the chunked results here
        }
    });

此外，如果您要返回一个 eloquent 对象，它将自动转换为 json，因此您不需要执行 json_encode() 就我而言'我知道了。

如何更快地检索 500k 记录的数据库数据？

How to retrieve 500k records DB data faster?

mysql

laravel

laravel-5.2