网络爬虫 SQLite3 SELECT for next link to crawl is crazy slow

Question

我正在 python 中制作网络爬虫，sqlite 查询有问题。查询有效但速度非常慢，有没有更好的方法来做到这一点？通常我对 sql 相当不错，但我真的遇到了所有这些参数的限制。

Table links 包含从网站 (href="LINK") 抓取的 links。下面的sql输出一个link接下来应该抓取

在特定时间内只能在一台主机上抓取 x 个网站。

SELECT *,
(SELECT COUNT(*) FROM links AS b WHERE b.hostname = a.hostname and b.lastCrawledTimestamp < ?)
AS hostLimitCount
FROM links AS a WHERE hostLimitCount < ? and lastCrawledTimestamp < ?
LIMIT 1

lastCrawledTimestamp 有一个 INDEX
主机名有一个 INDEX

谢谢！

Answer 1

子查询将受益于

INDEX(hostname, lastCrawledTimestame)

另外，像下面这样把它翻过来，可能会有帮助：

SELECT  a.*, x.hostLimitCount
    FROM  
        (
        SELECT  b.hostname, COUNT(*) AS hostLimitCount
            FROM  links AS b
            WHERE  b.lastCrawledTimestamp < ?
            GROUP BY hostname
            HAVING  hostLimitCount < ?
            LIMIT  1 
        ) AS x
    JOIN  links AS a USING(hostname)

可能会受益于相同的索引。

网络爬虫 SQLite3 SELECT for next link to crawl is crazy slow

Web crawler SQLite3 SELECT for next link to crawl is crazy slow

python

mysql

sqlite

performance

web-crawler