使用多处理创建数据并将其保存到 sql 数据库

Question

我正在尝试通过在单个进程中收集所有数据并将其插入到 sqlite 数据库来保存使用多处理创建的数据：

from multiprocessing import Process, Queue, Manager
import time
import sqlite3
import queue
import os

def make_up_numbers(q):
    for i in range(5):
        time.sleep(1)
        q.put([i]*3)

def save_numbers(q):
    con = sqlite3.connect('test')
    c = con.cursor()
    query = 'INSERT INTO "test" VALUES (?, ?, ?)'
    x = True
    while x:
        item = q.get(timeout = 3) 
        try:
            c.execute(query, item)
        except queue.Empty:
            con.commit()
            con.close()
            x = False

if os.path.isfile('test'):
    os.remove('test')
con = sqlite3.connect('test')
c = con.cursor()
c.execute('CREATE TABLE "test" ("col1" INTEGER, "col2" INTEGER, "col3" INTEGER)')
con.commit()
con.close()
 
q = Manager().Queue()   
o = []  
p = Process(target = save_numbers, args = (q, ), daemon = True)
p.start()
o.append(p)

for i in range(10):
    p = Process(target = make_up_numbers, args = (q, ))
    p.start()
    o.append(p)

for p in o:
    p.join()

然而，应该触发数据提交和循环结束的 queue.Empty 异常，而不是 returns 以下错误：

Process Process-179:
Traceback (most recent call last):
  File "/home/vincent/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/vincent/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-52-63a13637663b>", line 19, in show_number
    item = q.get(timeout = 3)
  File "<string>", line 2, in get
  File "/home/vincent/anaconda3/lib/python3.7/multiprocessing/managers.py", line 834, in _callmethod
    raise convert_to_error(kind, result)
_queue.Empty

谁能帮我解决这个问题？

Answer 1

将 q.get() 语句放在的 try: 块中 Queue.Empty。实际上，在执行到达 try 语句之前抛出异常。

使用多处理创建数据并将其保存到 sql 数据库

Create data with multiprocessing and save it to sql database

python

sqlite

queue

multiprocessing