我可以在等待子进程 Popen 时在另一个线程中工作吗

Question

我有一个 Python 3.7 项目

它正在使用一个库，该库使用子进程 Popen 调用 shell 脚本。

我想知道：如果将库调用放在一个单独的线程中，我是否可以在主线程中工作，同时在另一个线程中等待 Popen 的结果？

这里有一个答案说：

The way Python threads work with the GIL is with a simple counter. With every 100 byte codes executed the GIL is supposed to be released by the thread currently executing in order to give other threads a chance to execute code. This behavior is essentially broken in Python 2.7 because of the thread release/acquire mechanism. It has been fixed in Python 3.

无论哪种方式听起来对我想做的事情都不是特别有希望。听起来如果 "library calls" 线程在调用 Popen.wait 时没有达到 100 字节码触发点，那么它可能不会切换到我的另一个线程并且整个应用程序将等待子进程？

但是这个信息可能是错误的。

这是另一个答案说：

...the interpreter can always release the GIL; it will give it to some other thread after it has interpreted enough instructions, or automatically if it does some I/O. Note that since recent Python 3.x, the criteria is no longer based on the number of executed instructions, but on whether enough time has elapsed.

这听起来更有希望，因为与子进程的通信可能涉及 I/O，因此可能允许我的主线程的上下文切换能够同时进行。（或者可能只是等待 wait 的经过时间会导致上下文切换）

我知道 https://docs.python.org/3/library/asyncio-subprocess.html 明确解决了这个问题，但我调用的是仅使用普通 subprocess.Popen.

的第 3 方库

任何人都可以确认 "subprocess calls in a separate thread" 想法是否对我有用，特别是在 Python 3.7 中？

Answer 1

我有时间做实验，所以我会回答我自己的问题...

我设置了两个文件：

mainthread.py

#!/usr/bin/env python
import subprocess
import threading
import time


def run_busyproc():
    print(f'{time.time()} Starting busyprocess...')
    subprocess.run(["python", "busyprocess.py"])
    print(f'{time.time()} busyprocess done.')


if __name__ == "__main__":
    thread = threading.Thread(target=run_busyproc)
    print("Starting thread...")
    thread.start()
    while thread.is_alive():
        print(f"{time.time()} Main thread doing its thing...")
        time.sleep(0.5)
    print("Thread is done (?)")
    print("Exit main.")

和busyprocess.py:

#!/usr/bin/env python
from time import sleep


if __name__ == "__main__":
    for _ in range(100):
        print("Busy...")
        sleep(0.5)
    print("Done")

运行 mainthread.py 从命令行我可以看到存在您希望看到的上下文切换 - 主线程能够在等待结果的同时进行工作子进程：

Starting thread...
1555970578.20475 Main thread doing its thing...
1555970578.204679 Starting busyprocess...

Busy...
1555970578.710308 Main thread doing its thing...
Busy...
1555970579.2153869 Main thread doing its thing...
Busy...
1555970579.718168 Main thread doing its thing...
Busy...
1555970580.2231748 Main thread doing its thing...
Busy...
1555970580.726122 Main thread doing its thing...
Busy...
1555970628.009814 Main thread doing its thing...

Done
1555970628.512945 Main thread doing its thing...

1555970628.518155 busyprocess done.
Thread is done (?)
Exit main.

各位好消息，python线程有效:)

我可以在等待子进程 Popen 时在另一个线程中工作吗

Can I do work in another thread while waiting for subprocess Popen

multithreading

subprocess

gil

python-3.x