如何在 for 循环中使用 subprocess.Popen？

Question

我正在尝试使用 subprocess.Popen 到运行一个命令，我在 subprocess.run 上选择了这个，因为我需要将命令的输出打印到 shell.

我在 for 循环中调用命令，并为每次迭代修改环境，但是 python 只有运行第一个命令，而不是其他 6.

我很确定这是一个错误，因为我找不到其他类似的东西，但我想在提交“错误”之前先检查一下。

相关代码如下所示

input_file = Path("input.json").open()

for thread in threads:
    new_env = default_env
    new_env["OMP_NUM_THREADS"] = str(thread)
    print("Starting run for {} threads".format(thread))
    process = subprocess.Popen(
        benchmark_command, env=new_env, stdin=input_file, stdout=subprocess.PIPE)
    lines = []
    for line in process.stdout:
        print(line.decode(), end='')
    results.append(Result(lines, thread))
    print("Completed run for {} threads".format(thread))

当前输出是这样的

❯ python python/plot_results.py
Starting run for 1 threads
<expected output from running command>
Completed run for 1 threads
Starting run for 2 threads
Completed run for 2 threads
Starting run for 4 threads
Completed run for 4 threads
Starting run for 8 threads
Completed run for 8 threads
Starting run for 16 threads
Completed run for 16 threads
Starting run for 32 threads
Completed run for 32 threads
Starting run for 56 threads
Completed run for 56 threads

但应该是这样的

❯ python python/plot_results.py
Starting run for 1 threads
<expected output from running command>
Completed run for 1 threads
Starting run for 2 threads
<expected output from running command>
Completed run for 2 threads
Starting run for 4 threads
<expected output from running command>
Completed run for 4 threads
Starting run for 8 threads
<expected output from running command>
Completed run for 8 threads
Starting run for 16 threads
<expected output from running command>
Completed run for 16 threads
Starting run for 32 threads
<expected output from running command>
Completed run for 32 threads
Starting run for 56 threads
<expected output from running command>
Completed run for 56 threads

Answer 1

首先我要说的是，除非你的语言专业水平远超高级，或者你的问题性质极其罕见，否则你的问题不太可能是bug，而是您可能对语言的工作方式有一些误解。

在您的情况下，Popen 不会阻塞您的主线程。您的问题是主线程正在启动所有进程...并最终开始从 stdout 读取输出并打印出来。

简单的解决方案是使用 subprocess.check_output 等待命令完成，但它不允许 env 参数，因此我们将在调用后使用 communicate() 方法Popen，这也会阻塞你的主线程并等待进程终止：

for thread in threads:
    new_env = default_env
    new_env["OMP_NUM_THREADS"] = str(thread)
    print("Starting run for {} threads".format(thread))
    output = subprocess.Popen(
        benchmark_command, env=new_env, stdin=input_file, stdout=subprocess.PIPE)
    stdout, stderr = output.communicate()
    lines = []
    for line in stdout:
        print(line.decode(), end='')
    results.append(Result(lines, thread))
    print("Completed run for {} threads".format(thread))

Answer 2

根据@jasonharper 的建议，我需要为每次迭代创建一个新的打开文件对象。

input_file = Path("input.json")

for thread in threads:
    new_env = default_env
    new_env["OMP_NUM_THREADS"] = str(thread)
    print("Starting run for {} threads".format(thread))
    process = subprocess.Popen(
        benchmark_command, env=new_env, stdin=input_file.open(), stdout=subprocess.PIPE)
    lines = []
    for line in process.stdout:
        print(line.decode(), end='')
    results.append(Result(lines, thread))
    print("Completed run for {} threads".format(thread))

如何在 for 循环中使用 subprocess.Popen？

How can I use subprocess.Popen in a for loop?

python

benchmarking

subprocess

python-3.x