如何在函数内初始化并行独立进程？
How to initialize parallel independent process within function?
抱歉，如果标题很奇怪。让我解释一下。
假设有 handler.py:
import funcs
import requests
def initialize_calculate(data):
check_data(data)
funcs.calculate(data) # takes a lot of time like 30 minutes
print('Calculation launched')
requests.get('hostname', params={'func':'calculate', 'status':'launched'})
这里是 funcs.py:
import requests
def calculate(data):
result = make_calculations(data)
requests.get('hostname',params={'func':'calculate', 'status':'finished', 'result':result})
所以我想要的是处理程序可以在任何地方初始化另一个函数，但不会等到它结束，因为我想通知 client-side 该进程已启动，以及何时完成此进程它自己会在完成后发送结果。
如何启动独立进程并从 initialize_calculate 计算函数？
我想知道如果没有 non-native 库或框架是否可行。
您可以使用 multiprocessing 模块中的 Process class 来做到这一点。
这是一个例子：
from multiprocessing import Process
import requests
def calculate(data):
result = make_calculations(data)
requests.get('hostname',params={'func':'calculate', 'status':'finished', 'result':result})
def initialize_calculate(data):
check_data(data)
p = Process(target=calculate, args=(data,))
p.start()
print('Calculation launched')
requests.get('hostname', params={'func':'calculate', 'status':'launched'})
如果您不想使用像 daemonocle implementing a "well-behaved" Unix-Daemon 这样的第 3 方库，您可以
使用 subprocess.Popen() 创建一个独立的进程。另一种选择是修改 multiprocessing.Process 以防止在 parent 退出时 child 的 auto-joining。
subprocess.Popen()
使用 subprocess.Popen()，您可以通过从终端手动指定命令和参数来启动新进程。这意味着您需要使 funcs.py 或另一个文件成为 top-level 脚本，该脚本从 stdin 解析 string-arguments，然后使用这些参数调用 funcs.calculate()。
我将您的示例归结为本质，因此我们不必阅读太多代码。
funcs.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
import os
import sys
import time
import psutil # 3rd party for demo
def print_msg(msg):
print(f"[{time.ctime()}, pid: {os.getpid()}] --- {msg}")
def calculate(data, *args):
print_msg(f"parent pid: {psutil.Process().parent().pid}, start calculate()")
for _ in range(int(500e6)):
pass
print_msg(f"parent pid: {psutil.Process().parent().pid}, end calculate()")
if __name__ == '__main__':
if len(sys.argv) > 1:
calculate(*sys.argv[1:])
subp_main.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
if __name__ == '__main__':
import time
import logging
import subprocess
import multiprocessing as mp
import funcs
mp.log_to_stderr(logging.DEBUG)
filename = funcs.__file__
data = ("data", 42)
# in case filename is an executable you don't need "python" before `filename`:
subprocess.Popen(args=["python", filename, *[str(arg) for arg in data]])
time.sleep(1) # keep parent alive a bit longer for demo
funcs.print_msg(f"exiting")
对于测试很重要，运行 来自终端，例如不是 PyCharm-Run，因为它不会显示 child 打印的内容。在下面的最后一行中，您看到 child 进程的 parent-id 更改为 1 因为 child 在 [=100] 之后被 systemd (Ubuntu) 采用=]退出。
$> ./subp_main.py
[Fri Oct 23 20:14:44 2020, pid: 28650] --- parent pid: 28649, start calculate()
[Fri Oct 23 20:14:45 2020, pid: 28649] --- exiting
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
$> [Fri Oct 23 20:14:54 2020, pid: 28650] --- parent pid: 1, end calculate()
class OrphanProcess(multiprocessing.Process)
如果您要搜索更方便的东西，那么您不能按原样使用 high-level multiprocessing.Process，因为它不会让 parent 进程退出 before the child，正如你所要求的。当 parent 关闭时，常规 child-processes 要么加入（等待）要么终止（如果您为 Process 设置了 daemon 标志）。这仍然发生在 Python 内。请注意，daemon-flag 不会使进程成为 Unix-Daemon。命名是 confusion.
的一个有点频繁的来源
我将 multiprocessing.Process 子类化以关闭 auto-joining 并花一些时间研究源代码并观察 systemd/init 是否 zombies might become an issue. Because the modification turns off automatic joining in the parent, I recommend using "forkserver" as start-method for new processes on Unix (always a good idea if the parent is already multi-threaded) to prevent zombie-children from sticking around as long the parent is still running. When the parent process terminates, its child-zombies get eventually reaped。 运行 multiprocessing.log_to_stderr() 显示一切正常关闭，所以到目前为止似乎没有任何问题。
考虑这种方法是实验性的，但它可能比使用原始 os.fork() 到 re-invent 广泛 multiprocessing 机器的一部分要安全得多，只是为了添加这一功能。对于 child 中的 error-handling，写一个 try-except 块并记录到文件。
orphan.py
import multiprocessing.util
import multiprocessing.process as mpp
import multiprocessing as mp
__all__ = ['OrphanProcess']
class OrphanProcess(mp.Process):
"""Process which won't be joined by parent on parent shutdown."""
def start(self):
super().start()
mpp._children.discard(self)
def __del__(self):
# Finalizer won't `.join()` the child because we discarded it,
# so here last chance to reap a possible zombie from within Python.
# Otherwise systemd/init will reap eventually.
self.join(0)
orph_main.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
if __name__ == '__main__':
import time
import logging
import multiprocessing as mp
from orphan import OrphanProcess
from funcs import print_msg, calculate
mp.set_start_method("forkserver")
mp.log_to_stderr(logging.DEBUG)
p = OrphanProcess(target=calculate, args=("data", 42))
p.start()
time.sleep(1)
print_msg(f"exiting")
再次从终端测试以将 child 打印到标准输出。当在第二个提示上打印完所有内容后 shell 似乎挂起时，按回车键获得新提示。 parent-id 在这里保持不变，因为从 OS-point 的角度来看，parent 是 forkserver-process，而不是 orph_main.py 的初始 main-process .
$> ./orph_main.py
[INFO/MainProcess] created temp directory /tmp/pymp-bd75vnol
[INFO/OrphanProcess-1] child process calling self.run()
[Fri Oct 23 21:18:29 2020, pid: 30998] --- parent pid: 30997, start calculate()
[Fri Oct 23 21:18:30 2020, pid: 30995] --- exiting
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
$> [Fri Oct 23 21:18:38 2020, pid: 30998] --- parent pid: 30997, end calculate()
[INFO/OrphanProcess-1] process shutting down
[DEBUG/OrphanProcess-1] running all "atexit" finalizers with priority >= 0
[DEBUG/OrphanProcess-1] running the remaining "atexit" finalizers
[INFO/OrphanProcess-1] process exiting with exitcode 0
抱歉，如果标题很奇怪。让我解释一下。
假设有 handler.py:
import funcs
import requests
def initialize_calculate(data):
check_data(data)
funcs.calculate(data) # takes a lot of time like 30 minutes
print('Calculation launched')
requests.get('hostname', params={'func':'calculate', 'status':'launched'})
这里是 funcs.py:
import requests
def calculate(data):
result = make_calculations(data)
requests.get('hostname',params={'func':'calculate', 'status':'finished', 'result':result})
所以我想要的是处理程序可以在任何地方初始化另一个函数，但不会等到它结束，因为我想通知 client-side 该进程已启动，以及何时完成此进程它自己会在完成后发送结果。
如何启动独立进程并从 initialize_calculate 计算函数？
我想知道如果没有 non-native 库或框架是否可行。
您可以使用 multiprocessing 模块中的 Process class 来做到这一点。
这是一个例子：
from multiprocessing import Process
import requests
def calculate(data):
result = make_calculations(data)
requests.get('hostname',params={'func':'calculate', 'status':'finished', 'result':result})
def initialize_calculate(data):
check_data(data)
p = Process(target=calculate, args=(data,))
p.start()
print('Calculation launched')
requests.get('hostname', params={'func':'calculate', 'status':'launched'})
如果您不想使用像 daemonocle implementing a "well-behaved" Unix-Daemon 这样的第 3 方库，您可以
使用 subprocess.Popen() 创建一个独立的进程。另一种选择是修改 multiprocessing.Process 以防止在 parent 退出时 child 的 auto-joining。
subprocess.Popen()
使用 subprocess.Popen()，您可以通过从终端手动指定命令和参数来启动新进程。这意味着您需要使 funcs.py 或另一个文件成为 top-level 脚本，该脚本从 stdin 解析 string-arguments，然后使用这些参数调用 funcs.calculate()。
我将您的示例归结为本质，因此我们不必阅读太多代码。
funcs.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
import os
import sys
import time
import psutil # 3rd party for demo
def print_msg(msg):
print(f"[{time.ctime()}, pid: {os.getpid()}] --- {msg}")
def calculate(data, *args):
print_msg(f"parent pid: {psutil.Process().parent().pid}, start calculate()")
for _ in range(int(500e6)):
pass
print_msg(f"parent pid: {psutil.Process().parent().pid}, end calculate()")
if __name__ == '__main__':
if len(sys.argv) > 1:
calculate(*sys.argv[1:])
subp_main.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
if __name__ == '__main__':
import time
import logging
import subprocess
import multiprocessing as mp
import funcs
mp.log_to_stderr(logging.DEBUG)
filename = funcs.__file__
data = ("data", 42)
# in case filename is an executable you don't need "python" before `filename`:
subprocess.Popen(args=["python", filename, *[str(arg) for arg in data]])
time.sleep(1) # keep parent alive a bit longer for demo
funcs.print_msg(f"exiting")
对于测试很重要，运行 来自终端，例如不是 PyCharm-Run，因为它不会显示 child 打印的内容。在下面的最后一行中，您看到 child 进程的 parent-id 更改为 1 因为 child 在 [=100] 之后被 systemd (Ubuntu) 采用=]退出。
$> ./subp_main.py
[Fri Oct 23 20:14:44 2020, pid: 28650] --- parent pid: 28649, start calculate()
[Fri Oct 23 20:14:45 2020, pid: 28649] --- exiting
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
$> [Fri Oct 23 20:14:54 2020, pid: 28650] --- parent pid: 1, end calculate()
class OrphanProcess(multiprocessing.Process)
如果您要搜索更方便的东西，那么您不能按原样使用 high-level multiprocessing.Process，因为它不会让 parent 进程退出 before the child，正如你所要求的。当 parent 关闭时，常规 child-processes 要么加入（等待）要么终止（如果您为 Process 设置了 daemon 标志）。这仍然发生在 Python 内。请注意，daemon-flag 不会使进程成为 Unix-Daemon。命名是 confusion.
我将 multiprocessing.Process 子类化以关闭 auto-joining 并花一些时间研究源代码并观察 systemd/init 是否 zombies might become an issue. Because the modification turns off automatic joining in the parent, I recommend using "forkserver" as start-method for new processes on Unix (always a good idea if the parent is already multi-threaded) to prevent zombie-children from sticking around as long the parent is still running. When the parent process terminates, its child-zombies get eventually reaped。 运行 multiprocessing.log_to_stderr() 显示一切正常关闭，所以到目前为止似乎没有任何问题。
考虑这种方法是实验性的，但它可能比使用原始 os.fork() 到 re-invent 广泛 multiprocessing 机器的一部分要安全得多，只是为了添加这一功能。对于 child 中的 error-handling，写一个 try-except 块并记录到文件。
orphan.py
import multiprocessing.util
import multiprocessing.process as mpp
import multiprocessing as mp
__all__ = ['OrphanProcess']
class OrphanProcess(mp.Process):
"""Process which won't be joined by parent on parent shutdown."""
def start(self):
super().start()
mpp._children.discard(self)
def __del__(self):
# Finalizer won't `.join()` the child because we discarded it,
# so here last chance to reap a possible zombie from within Python.
# Otherwise systemd/init will reap eventually.
self.join(0)
orph_main.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
if __name__ == '__main__':
import time
import logging
import multiprocessing as mp
from orphan import OrphanProcess
from funcs import print_msg, calculate
mp.set_start_method("forkserver")
mp.log_to_stderr(logging.DEBUG)
p = OrphanProcess(target=calculate, args=("data", 42))
p.start()
time.sleep(1)
print_msg(f"exiting")
再次从终端测试以将 child 打印到标准输出。当在第二个提示上打印完所有内容后 shell 似乎挂起时，按回车键获得新提示。 parent-id 在这里保持不变，因为从 OS-point 的角度来看，parent 是 forkserver-process，而不是 orph_main.py 的初始 main-process .
$> ./orph_main.py
[INFO/MainProcess] created temp directory /tmp/pymp-bd75vnol
[INFO/OrphanProcess-1] child process calling self.run()
[Fri Oct 23 21:18:29 2020, pid: 30998] --- parent pid: 30997, start calculate()
[Fri Oct 23 21:18:30 2020, pid: 30995] --- exiting
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
$> [Fri Oct 23 21:18:38 2020, pid: 30998] --- parent pid: 30997, end calculate()
[INFO/OrphanProcess-1] process shutting down
[DEBUG/OrphanProcess-1] running all "atexit" finalizers with priority >= 0
[DEBUG/OrphanProcess-1] running the remaining "atexit" finalizers
[INFO/OrphanProcess-1] process exiting with exitcode 0