Popen / Wait - 等待永远不会结束

Question

import sys
import os
from subprocess import Popen, PIPE, STDOUT

# Transfer Database
print ('Transferring from ' + mysql_source_database)
mysql = Popen(f"mysql -h {mysql_dest_host} -P 3306 -u {mysql_dest_username} -p{mysql_dest_pw} {mysql_dest_database}".split(), stdin=PIPE, stdout=PIPE)
dbnamerewrite = Popen(f"sed s/{mysql_source_database}/{mysql_dest_database}/g".split(), stdin=PIPE, stdout=mysql.stdin)
mysqldump = Popen(f"mysqldump --set-gtid-purged=OFF --column-statistics=0 -h {mysql_source_host} -P 3306 -u {mysql_source_username} -p{mysql_source_pw} {mysql_source_database}".split(), stdout=dbnamerewrite.stdin)
mysql_stdout = mysql.communicate()[0]
mysqldump.wait()

上面的代码完成了我想要的，但从未停止等待。有谁知道如何解决等待。如果我在 SQL 工作完成后按 ctrl-c，这是给定的错误：

^CERROR 1064 (42000) at line 3829: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 3
Traceback (most recent call last):
  File "test.py", line 19, in <module>
    mysql_stdout = mysql.communicate()[0]
  File "/usr/lib/python3.8/subprocess.py", line 1028, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.8/subprocess.py", line 1868, in _communicate
    ready = selector.select(timeout)
  File "/usr/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt

Answer 1

一件事是您应该放弃对 mysqldump.wait() 的显式调用。根据 docs:

Note: This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.

mysql.communicate 在这种情况下就足够了，因为它不会收到 EOF，直到管道上的所有元素都发送一个 EOF。所以mysql.communicate()直接返回意味着其他两个过程已经完成。

另一个问题是，根据您拥有的进程的顺序，您必须以相反的顺序对所有进程调用 communicate 才能让数据流经管道。一种解决方案就是这样做：

db_param = ['-h', mysql_dest_host, '-P', '3306', '-u', mysql_dest_username, f'p{mysql_dest_pw}', mysql_dest_database]

mysql = Popen(['mysql'] + db_param,
              stdin=PIPE, stdout=PIPE)
dbnamerewrite = Popen(['sed', f's/{mysql_source_database}/{mysql_dest_database}/g'],
                      stdin=PIPE, stdout=mysql.stdin)
mysqldump = Popen(['mysqldump', '--set-gtid-purged=OFF', '--column-statistics=0'] + db_param,
                  stdout=dbnamerewrite.stdin)

mysqldump.communicate()
dbnamerewrite.communicate()
mysql_stdout = mysql.communicate()[0]

另一种选择是以相反的顺序设置您的管道，在这种情况下您只需要 communicate 最后一个过程：

db_param = ['-h', mysql_dest_host, '-P', '3306', '-u', mysql_dest_username, f'p{mysql_dest_pw}', mysql_dest_database]

mysqldump = Popen(['mysqldump', '--set-gtid-purged=OFF', '--column-statistics=0'] + db_param,
                  stdout=PIPE)
dbnamerewrite = Popen(['sed', f's/{mysql_source_database}/{mysql_dest_database}/g'],
                      stdin=mysqldump.stdout, stdout=PIPE)
mysql = Popen(['mysql'] + db_param, stdin=dbnamerewrite.stdout, stdout=PIPE)

mysql_stdout = mysql.communicate()[0]

Popen / Wait - 等待永远不会结束

Popen / Wait - Wait never finishes

python

subprocess

python-3.x