将 SIGTERM 发送到 运行 任务,分布式任务
Send SIGTERM to the running task, dask distributed
当我将小型 Tensorflow 训练作为单个任务提交时,它会启动额外的线程。当我按下 Ctrl+C
并提高 KeyboardInterrupt
时,我的任务已关闭,但底层线程未清理,训练继续进行。
最初,我认为这是 Tensorflow 的问题(不是清理线程),但经过测试,我了解到问题来自 Dask 端,可能不会将 SIGTERM 信号进一步填充到任务中功能。我的问题是,如何设置 Dask 以将 SIGTERM 信号填充到 运行 任务?
所需流量示例:
本地进程 -> 按 Ctrl + C -> Dask 调度程序 -> Dask worker -> SIGTERM 信号 -> 运行 使用 Tensorflow 训练的单任务。
谢谢。
P.S 如果您需要更多信息,请询问。
更新:
代码示例:
c = Client('<remote-scheduler>')
def task():
# tensorflow training
model = ...
model.fit(x_train, y_train)
training = c.submit(task)
training.result()
现在,在训练期间,当我按下 Ctrl+C
时任务被取消,但 tensorflow threads/processes 仍然存在。
更新 2:
ps -f -u [username]
命令输出。
Dask 集群(1 个调度程序、1 个工作人员、同一台服务器),没有 运行 个任务:
UID PID PPID C STIME TTY TIME CMD
vladysl+ 16547 1 0 12:40 ? 00:00:00 /lib/systemd/systemd --user
vladysl+ 16550 16547 0 12:40 ? 00:00:00 (sd-pam)
vladysl+ 16805 16311 0 12:40 ? 00:00:00 sshd: vladyslav@pts/45
vladysl+ 16811 16805 0 12:40 pts/45 00:00:00 -bash
vladysl+ 18946 16811 4 12:41 pts/45 00:00:24 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-scheduler --port 42001
vladysl+ 22284 22175 0 12:46 ? 00:00:00 sshd: vladyslav@pts/38
vladysl+ 22285 22284 0 12:46 pts/38 00:00:00 -bash
vladysl+ 23138 16811 1 12:48 pts/45 00:00:03 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-worker localhost:42001 --worker-port 420011 --memory-limit $
vladysl+ 23143 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(11)
vladysl+ 23145 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23151 23145 99 12:48 pts/45 00:03:48 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23536 23151 0 12:49 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(25)
vladysl+ 26150 22285 0 12:51 pts/38 00:00:00 ps -f -u vladyslav
任务期间运行:
UID PID PPID C STIME TTY TIME CMD
vladysl+ 16547 1 0 12:40 ? 00:00:00 /lib/systemd/systemd --user
vladysl+ 16811 16805 0 12:40 pts/45 00:00:00 -bash
vladysl+ 18946 16811 4 12:41 pts/45 00:00:30 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-scheduler --port 42001
vladysl+ 22285 22284 0 12:46 pts/38 00:00:00 -bash
vladysl+ 23138 16811 1 12:48 pts/45 00:00:06 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-worker localhost:42001 --worker-port 420011 --memory-limit $
vladysl+ 23143 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(11)
vladysl+ 23145 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23151 23145 99 12:48 pts/45 00:07:55 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23536 23151 0 12:49 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(25)
vladysl+ 27079 22285 0 12:54 pts/38 00:00:00 ps -f -u vladyslav
按下 Ctrl+C
后,任务取消但 tensorflow 继续工作:
UID PID PPID C STIME TTY TIME CMD
vladysl+ 16811 16805 0 12:40 pts/45 00:00:00 -bash
vladysl+ 18946 16811 4 12:41 pts/45 00:00:31 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-scheduler --port 42001
vladysl+ 22285 22284 0 12:46 pts/38 00:00:00 -bash
vladysl+ 23138 16811 1 12:48 pts/45 00:00:06 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-worker localhost:42001 --worker-port 420011 --memory-limit $
vladysl+ 23143 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(11)
vladysl+ 23145 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23151 23145 99 12:48 pts/45 00:09:32 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23536 23151 0 12:49 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(25)
vladysl+ 27117 22285 0 12:54 pts/38 00:00:00 ps -f -u vladyslav
如您所见,没有出现任何新内容。
Dask 不支持从客户端向工作人员 运行 任务传播信号。
当我将小型 Tensorflow 训练作为单个任务提交时,它会启动额外的线程。当我按下 Ctrl+C
并提高 KeyboardInterrupt
时,我的任务已关闭,但底层线程未清理,训练继续进行。
最初,我认为这是 Tensorflow 的问题(不是清理线程),但经过测试,我了解到问题来自 Dask 端,可能不会将 SIGTERM 信号进一步填充到任务中功能。我的问题是,如何设置 Dask 以将 SIGTERM 信号填充到 运行 任务?
所需流量示例:
本地进程 -> 按 Ctrl + C -> Dask 调度程序 -> Dask worker -> SIGTERM 信号 -> 运行 使用 Tensorflow 训练的单任务。
谢谢。
P.S 如果您需要更多信息,请询问。
更新:
代码示例:
c = Client('<remote-scheduler>')
def task():
# tensorflow training
model = ...
model.fit(x_train, y_train)
training = c.submit(task)
training.result()
现在,在训练期间,当我按下 Ctrl+C
时任务被取消,但 tensorflow threads/processes 仍然存在。
更新 2:
ps -f -u [username]
命令输出。
Dask 集群(1 个调度程序、1 个工作人员、同一台服务器),没有 运行 个任务:
UID PID PPID C STIME TTY TIME CMD
vladysl+ 16547 1 0 12:40 ? 00:00:00 /lib/systemd/systemd --user
vladysl+ 16550 16547 0 12:40 ? 00:00:00 (sd-pam)
vladysl+ 16805 16311 0 12:40 ? 00:00:00 sshd: vladyslav@pts/45
vladysl+ 16811 16805 0 12:40 pts/45 00:00:00 -bash
vladysl+ 18946 16811 4 12:41 pts/45 00:00:24 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-scheduler --port 42001
vladysl+ 22284 22175 0 12:46 ? 00:00:00 sshd: vladyslav@pts/38
vladysl+ 22285 22284 0 12:46 pts/38 00:00:00 -bash
vladysl+ 23138 16811 1 12:48 pts/45 00:00:03 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-worker localhost:42001 --worker-port 420011 --memory-limit $
vladysl+ 23143 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(11)
vladysl+ 23145 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23151 23145 99 12:48 pts/45 00:03:48 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23536 23151 0 12:49 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(25)
vladysl+ 26150 22285 0 12:51 pts/38 00:00:00 ps -f -u vladyslav
任务期间运行:
UID PID PPID C STIME TTY TIME CMD
vladysl+ 16547 1 0 12:40 ? 00:00:00 /lib/systemd/systemd --user
vladysl+ 16811 16805 0 12:40 pts/45 00:00:00 -bash
vladysl+ 18946 16811 4 12:41 pts/45 00:00:30 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-scheduler --port 42001
vladysl+ 22285 22284 0 12:46 pts/38 00:00:00 -bash
vladysl+ 23138 16811 1 12:48 pts/45 00:00:06 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-worker localhost:42001 --worker-port 420011 --memory-limit $
vladysl+ 23143 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(11)
vladysl+ 23145 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23151 23145 99 12:48 pts/45 00:07:55 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23536 23151 0 12:49 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(25)
vladysl+ 27079 22285 0 12:54 pts/38 00:00:00 ps -f -u vladyslav
按下 Ctrl+C
后,任务取消但 tensorflow 继续工作:
UID PID PPID C STIME TTY TIME CMD
vladysl+ 16811 16805 0 12:40 pts/45 00:00:00 -bash
vladysl+ 18946 16811 4 12:41 pts/45 00:00:31 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-scheduler --port 42001
vladysl+ 22285 22284 0 12:46 pts/38 00:00:00 -bash
vladysl+ 23138 16811 1 12:48 pts/45 00:00:06 /home/vladyslav/miniconda3/envs/py3.6/bin/python /home/vladyslav/miniconda3/envs/py3.6/bin/dask-worker localhost:42001 --worker-port 420011 --memory-limit $
vladysl+ 23143 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(11)
vladysl+ 23145 23138 0 12:48 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23151 23145 99 12:48 pts/45 00:09:32 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.forkserver import main; main(15, 16, ['distributed'], **{'sys_path': ['/home/vlady$
vladysl+ 23536 23151 0 12:49 pts/45 00:00:00 /home/vladyslav/miniconda3/envs/py3.6/bin/python -c from multiprocessing.semaphore_tracker import main;main(25)
vladysl+ 27117 22285 0 12:54 pts/38 00:00:00 ps -f -u vladyslav
如您所见,没有出现任何新内容。
Dask 不支持从客户端向工作人员 运行 任务传播信号。