Python 循环中的多线程执行查询 sql

Question

我想在多线程中改造我的程序，但我没有说怎么做。事实上，我想到了 2 种解决方案，但我认为这不是最优的：执行 select 并将结果存储在 python 中的数组中，然后按线程范围过滤。或带有子句 ID 的多线程 select 线程只有一个域有没有更好的方法直接在循环中做这个？

#!/usr/bin/python
import mysql.connector as mariadb
from subprocess import Popen, PIPE, STDOUT
import re
import os

mariadb_connection = mariadb.connect(user='root', password='xxxx', database='xxxx');
cursor = mariadb_connection.cursor(buffered=True)
#retrieving information

cursor.execute("SELECT Domain,Id FROM classement where Domain like '%com';")

for Domain,Id in cursor:
          counter=0
          for character in Domain:
                if (character == "."):
                        counter = counter + 1
                        if (counter==1):
                            print(Domain)
                            pingresult = Popen(['ping','-c','3', Domain], stdout=PIPE, stderr=STDOUT).communicate()[0]
                            pingresult=str(pingresult)
                            ping = pingresult.find('min/avg/max/mdev')
                            ping=pingresult[ping+19:-6]
                            print(ping)
                            cursor.execute('update classement set Ping = "%s" where Domain = "%s";' % (ping,Domain))
                            mariadb_connection.commit()
mariadb_connection.close()

Answer 1

我认为最简单的方法是将循环包装在函数中并使用 multiprocessing.dummy 线程池：

from multiprocessing.dummy import Pool
...
cursor.execute( ... )
...

def process (entry):
    Domain,Id = entry
    ...
    # your code there
    ...
    cursor.execute( ... )
    mariadb_connection.commit()

pool = Pool(os.cpu_count())
# or may be N*os.cpu_count() if ping waits for
# network interaction too long

pool.map(process, cursor)
pool.join()

多线程访问数据库可能需要额外的锁定。

Python 循环中的多线程执行查询 sql

Python multithreading in a loop for executing query sql

python

multithreading

python-multiprocessing