无法在 Python 多处理中获取参数

Question

import re, os
from multiprocessing import Pool

directory = r'F:\data\' #['a.txt', 'b.txt', 'c.txt', 'd.txt', 'e.txt']
raw_files = os.listdir(directory)

# TARGET FUNCTION
def print_result(raw_files):
    for raw_file in raw_files:
        with open(directory+raw_file, 'r', encoding = 'utf-16') as f: #FileNotFoundError: [Errno 2] No such file or directory: 'F:\corpus_duplicated\\2'
            raw = f.read()
            if re.search('target', raw):
                print(raw)

if __name__ == '__main__':
    print(raw_files[:3]) #['a.txt', 'b.txt', 'c.txt']
    pool = Pool(processes = 4)
    pool.map(print_result, raw_files)
    pool.close()
    pool.join()

我想directory+raw_file成为F:\data\a.txt 但它导致 F:\data 可以在错误消息中看到。

我想我还不了解多处理，但我无法通过搜索了解原因。

感谢您的帮助。（附上代码减一。）

Answer 1

应该更改函数 print_result，使其只处理一个文件。您给 map 包含文件的列表，它会将列表中的项目分块，然后将它们分块交给 print_result

import re, os
from multiprocessing import Pool

directory = 'F:/data/'
raw_files = os.listdir(directory)

# TARGET FUNCTION
# input to this function should be a single file
def print_result(raw_file):  
    with open(directory+raw_file, 'r', encoding = 'utf-16') as f:
        raw = f.read()
        if re.search('target', raw):
            print(raw)

if __name__ == '__main__':
    print(raw_files[:3]) #['a.txt', 'b.txt', 'c.txt']
    pool = Pool(processes = 4)
    pool.map(print_result, raw_files)
    pool.close()
    pool.join()

您可能需要将 directory

的正斜杠改回反斜杠

无法在 Python 多处理中获取参数

Cannot get arguments in Python Multiprocessing

python

multiprocessing

python-multiprocessing