如何将未腌制的对象作为 multiprocessing.Process 中的参数传递？

Question

我正在解开一个对象并作为参数之一传递 process.Getting 无输出。

我想知道这种 unpickling 并将对象作为参数传递的方法是否会导致多处理错误。解决这个问题的方法是什么？

from multiprocessing import Process, Queue
def func(arg1, arg2,q):
    df_temp = arg1[arg1['col'].isin(arg2)]
    q.put(df_temp)

if __name__ == '__main__':
    import pickle
    import pandas as pd
    arg1= pickle.load(open('paths.p','rb'))
    arg2= pd.Series(pd.date_range(end = 'some_Date', periods=12,freq = 'MS')).dt.to_pydatetime()
    arg2=[i.date() for i in arg2]
    q = Queue()
    p =Process(target=func, args=(arg1,arg2,q))
    p.start()
    p.join()    
    while not q.empty():
        w=q.get()

Answer 1

你因为其他原因陷入僵局。

By default if a process is not the creator of the queue then on exit it will attempt to join the queue’s background thread. The process can call cancel_join_thread() to make join_thread() do nothing. docs

您的 Process 不会退出，因为它正在加入来自 multiprocessing.Queue 的供稿线程，它会在您第一次 queue.put() 时立即启动。您需要在 parent queue.get() 之前 加入流程。

Warning: As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe. This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children. docs

也不要使用while not q.empty()，它是一个anti-pattern，一旦你有多个消费者就会导致死锁。请改用 sentinel-values 来通知消费者没有其他商品即将到来。有关更多信息，。

如何将未腌制的对象作为 multiprocessing.Process 中的参数传递？

How to pass unpickled object as argument in multiprocessing.Process?

python

queue

pickle

multiprocessing

python-multiprocessing