celery with redis：混合具有不同并发性的工人
celery with redis: mixing workers with different concurrency
我使用以下脚本将任务分配给不同节点上的两个工作人员。
celery_call.py:
from celery_test import add
import time
results = []
for i in range(20):
results.append(add.delay(i))
for result in results:
timeStart = time.time()
resultValue = result.get(timeout=10)
timePassed = time.time() - timeStart
print(timePassed, resultValue)
celery_test.py:
from celery import Celery
app = Celery('celery_test', backend='redis://ip', broker='redis://ip')
@app.task
def add(x):
import time
time.sleep(2)
return x
我有两个工人 运行 在两个不同的节点上。第一个节点：
celery -A celery_test worker --concurrency 1 -l INFO
和第二个节点：
celery -A celery_test worker --concurrency 10 -l INFO
任务在几秒钟内分配并解决。
在第一个节点上的 10 秒内：
[2017-05-27 13:46:22,529: INFO/MainProcess] Received task: celery_test.add[2d9d592f-391f-4e1f-8dd2-5e50e5977c81]
[2017-05-27 13:46:22,531: INFO/MainProcess] Received task: celery_test.add[e5c20ae9-92d7-4811-9b54-4efd3707bb5f]
[2017-05-27 13:46:22,533: INFO/MainProcess] Received task: celery_test.add[bb9b50bc-1c5d-4ede-abfb-9591f53b3912]
[2017-05-27 13:46:22,535: INFO/MainProcess] Received task: celery_test.add[b195fbb4-2683-461e-aee7-4715a1387eb6]
[2017-05-27 13:46:22,537: INFO/MainProcess] Received task: celery_test.add[c8ce9d10-f03a-4585-9ff8-0d49cfd2e8b2]
[2017-05-27 13:46:24,538: INFO/PoolWorker-1] Task celery_test.add[2d9d592f-391f-4e1f-8dd2-5e50e5977c81] succeeded in 2.0077112819999456s: 1
[2017-05-27 13:46:26,543: INFO/PoolWorker-1] Task celery_test.add[e5c20ae9-92d7-4811-9b54-4efd3707bb5f] succeeded in 2.0030374974012375s: 3
[2017-05-27 13:46:28,547: INFO/PoolWorker-1] Task celery_test.add[bb9b50bc-1c5d-4ede-abfb-9591f53b3912] succeeded in 2.0030434764921665s: 5
[2017-05-27 13:46:30,551: INFO/PoolWorker-1] Task celery_test.add[b195fbb4-2683-461e-aee7-4715a1387eb6] succeeded in 2.0029842611402273s: 7
[2017-05-27 13:46:32,555: INFO/PoolWorker-1] Task celery_test.add[c8ce9d10-f03a-4585-9ff8-0d49cfd2e8b2] succeeded in 2.0029691718518734s: 9
在第二个节点上的 4 秒内：
[2017-05-27 13:46:22,528: INFO/MainProcess] Received task: celery_test.add[402fb858-d9ec-4565-ab71-fbf4ec531787]
[2017-05-27 13:46:22,530: INFO/MainProcess] Received task: celery_test.add[311fc7ed-e44a-4119-a0fa-c6849574723e]
[2017-05-27 13:46:22,532: INFO/MainProcess] Received task: celery_test.add[af54e423-651b-4b01-a3d1-26ead5ae6af1]
[2017-05-27 13:46:22,534: INFO/MainProcess] Received task: celery_test.add[29234f7f-f841-44c2-94fb-b491a074318b]
[2017-05-27 13:46:22,537: INFO/MainProcess] Received task: celery_test.add[1a638710-810a-422d-8f2f-d554af3c5a92]
[2017-05-27 13:46:22,538: INFO/MainProcess] Received task: celery_test.add[5b1a6863-1b62-4927-849b-e04e2d34ce7c]
[2017-05-27 13:46:22,540: INFO/MainProcess] Received task: celery_test.add[f4cd393a-2f02-48dd-b27f-d73806e154da]
[2017-05-27 13:46:22,543: INFO/MainProcess] Received task: celery_test.add[da8241bf-ce4e-4fe6-bd65-91350da8b163]
[2017-05-27 13:46:22,544: INFO/MainProcess] Received task: celery_test.add[4892b31a-e488-4011-86e7-d55eb941cf1f]
[2017-05-27 13:46:22,545: INFO/MainProcess] Received task: celery_test.add[c883b6ec-3842-4b50-bfab-35613f1724ed]
[2017-05-27 13:46:22,548: INFO/MainProcess] Received task: celery_test.add[1b021f4c-b41d-46a8-8548-7016539e8a8b]
[2017-05-27 13:46:22,549: INFO/MainProcess] Received task: celery_test.add[ae5d9b7d-0fa2-493b-aa0b-0b13b3764fdf]
[2017-05-27 13:46:22,551: INFO/MainProcess] Received task: celery_test.add[af0d27fe-394f-4fbb-821e-95acbb99324c]
[2017-05-27 13:46:22,552: INFO/MainProcess] Received task: celery_test.add[0b91cc9d-63c4-4a39-9d03-35ac474252bc]
[2017-05-27 13:46:22,553: INFO/MainProcess] Received task: celery_test.add[1be5a881-064c-4cf8-8a18-10c2fe0400ed]
[2017-05-27 13:46:24,538: INFO/PoolWorker-1] Task celery_test.add[402fb858-d9ec-4565-ab71-fbf4ec531787] succeeded in 2.0080740470439196s: 0
[2017-05-27 13:46:24,540: INFO/PoolWorker-3] Task celery_test.add[311fc7ed-e44a-4119-a0fa-c6849574723e] succeeded in 2.007692627608776s: 2
[2017-05-27 13:46:24,543: INFO/PoolWorker-6] Task celery_test.add[29234f7f-f841-44c2-94fb-b491a074318b] succeeded in 2.007782282307744s: 6
[2017-05-27 13:46:24,543: INFO/PoolWorker-5] Task celery_test.add[af54e423-651b-4b01-a3d1-26ead5ae6af1] succeeded in 2.007910629734397s: 4
[2017-05-27 13:46:24,546: INFO/PoolWorker-8] Task celery_test.add[1a638710-810a-422d-8f2f-d554af3c5a92] succeeded in 2.0075698774307966s: 8
[2017-05-27 13:46:24,548: INFO/PoolWorker-2] Task celery_test.add[f4cd393a-2f02-48dd-b27f-d73806e154da] succeeded in 2.0072230715304613s: 11
[2017-05-27 13:46:24,548: INFO/PoolWorker-10] Task celery_test.add[5b1a6863-1b62-4927-849b-e04e2d34ce7c] succeeded in 2.007256705313921s: 10
[2017-05-27 13:46:24,552: INFO/PoolWorker-7] Task celery_test.add[da8241bf-ce4e-4fe6-bd65-91350da8b163] succeeded in 2.0082139261066914s: 12
[2017-05-27 13:46:24,554: INFO/PoolWorker-9] Task celery_test.add[c883b6ec-3842-4b50-bfab-35613f1724ed] succeeded in 2.0077442210167646s: 14
[2017-05-27 13:46:24,554: INFO/PoolWorker-4] Task celery_test.add[4892b31a-e488-4011-86e7-d55eb941cf1f] succeeded in 2.007783567532897s: 13
[2017-05-27 13:46:26,542: INFO/PoolWorker-1] Task celery_test.add[1b021f4c-b41d-46a8-8548-7016539e8a8b] succeeded in 2.002950184047222s: 15
[2017-05-27 13:46:26,544: INFO/PoolWorker-3] Task celery_test.add[ae5d9b7d-0fa2-493b-aa0b-0b13b3764fdf] succeeded in 2.002891855314374s: 16
[2017-05-27 13:46:26,547: INFO/PoolWorker-5] Task celery_test.add[af0d27fe-394f-4fbb-821e-95acbb99324c] succeeded in 2.002899706363678s: 17
[2017-05-27 13:46:26,547: INFO/PoolWorker-6] Task celery_test.add[0b91cc9d-63c4-4a39-9d03-35ac474252bc] succeeded in 2.002899182960391s: 18
[2017-05-27 13:46:26,550: INFO/PoolWorker-8] Task celery_test.add[1be5a881-064c-4cf8-8a18-10c2fe0400ed] succeeded in 2.0029856264591217s: 19
但是，检索结果会延迟，总共需要 20 秒：
1.9911870956420898 0
0.0006098747253417969 1
0.0011210441589355469 2
2.003366231918335 3
0.0006439685821533203 4
2.0034918785095215 5
1.0012683868408203 6
1.00254487991333 7
1.001213788986206 8
1.002840518951416 9
1.0012362003326416 10
1.001204490661621 11
1.00126314163208 12
1.0012261867523193 13
1.0012249946594238 14
1.0012695789337158 15
1.0013458728790283 16
1.0013868808746338 17
1.0014445781707764 18
1.001399278640747 19
我有两个问题：
如何设置 celery 以优化工作分配？例如。一个好的解决方案是将 1+1 个任务分配给第一个节点，将 10+8 个任务分配给第二个节点。
恕我直言，检索结果的时间应该与最慢的工作人员一样长，即 10 秒。为什么需要更长的时间？如何加速？
关闭第一个worker，第二个worker 10+10个任务需要4秒（ok），取回结果需要5秒。为什么我还松了一秒？
Understanding celery task prefetching 将运行时间减少到 4 或 5 秒。还是有时候1秒太多，但是1.和2.都解决了。
由于这似乎是一个不同的问题，所以我在一个单独的问题中再次询问了 3.：
我使用以下脚本将任务分配给不同节点上的两个工作人员。
celery_call.py:
from celery_test import add
import time
results = []
for i in range(20):
results.append(add.delay(i))
for result in results:
timeStart = time.time()
resultValue = result.get(timeout=10)
timePassed = time.time() - timeStart
print(timePassed, resultValue)
celery_test.py:
from celery import Celery
app = Celery('celery_test', backend='redis://ip', broker='redis://ip')
@app.task
def add(x):
import time
time.sleep(2)
return x
我有两个工人 运行 在两个不同的节点上。第一个节点：
celery -A celery_test worker --concurrency 1 -l INFO
和第二个节点：
celery -A celery_test worker --concurrency 10 -l INFO
任务在几秒钟内分配并解决。 在第一个节点上的 10 秒内：
[2017-05-27 13:46:22,529: INFO/MainProcess] Received task: celery_test.add[2d9d592f-391f-4e1f-8dd2-5e50e5977c81]
[2017-05-27 13:46:22,531: INFO/MainProcess] Received task: celery_test.add[e5c20ae9-92d7-4811-9b54-4efd3707bb5f]
[2017-05-27 13:46:22,533: INFO/MainProcess] Received task: celery_test.add[bb9b50bc-1c5d-4ede-abfb-9591f53b3912]
[2017-05-27 13:46:22,535: INFO/MainProcess] Received task: celery_test.add[b195fbb4-2683-461e-aee7-4715a1387eb6]
[2017-05-27 13:46:22,537: INFO/MainProcess] Received task: celery_test.add[c8ce9d10-f03a-4585-9ff8-0d49cfd2e8b2]
[2017-05-27 13:46:24,538: INFO/PoolWorker-1] Task celery_test.add[2d9d592f-391f-4e1f-8dd2-5e50e5977c81] succeeded in 2.0077112819999456s: 1
[2017-05-27 13:46:26,543: INFO/PoolWorker-1] Task celery_test.add[e5c20ae9-92d7-4811-9b54-4efd3707bb5f] succeeded in 2.0030374974012375s: 3
[2017-05-27 13:46:28,547: INFO/PoolWorker-1] Task celery_test.add[bb9b50bc-1c5d-4ede-abfb-9591f53b3912] succeeded in 2.0030434764921665s: 5
[2017-05-27 13:46:30,551: INFO/PoolWorker-1] Task celery_test.add[b195fbb4-2683-461e-aee7-4715a1387eb6] succeeded in 2.0029842611402273s: 7
[2017-05-27 13:46:32,555: INFO/PoolWorker-1] Task celery_test.add[c8ce9d10-f03a-4585-9ff8-0d49cfd2e8b2] succeeded in 2.0029691718518734s: 9
在第二个节点上的 4 秒内：
[2017-05-27 13:46:22,528: INFO/MainProcess] Received task: celery_test.add[402fb858-d9ec-4565-ab71-fbf4ec531787]
[2017-05-27 13:46:22,530: INFO/MainProcess] Received task: celery_test.add[311fc7ed-e44a-4119-a0fa-c6849574723e]
[2017-05-27 13:46:22,532: INFO/MainProcess] Received task: celery_test.add[af54e423-651b-4b01-a3d1-26ead5ae6af1]
[2017-05-27 13:46:22,534: INFO/MainProcess] Received task: celery_test.add[29234f7f-f841-44c2-94fb-b491a074318b]
[2017-05-27 13:46:22,537: INFO/MainProcess] Received task: celery_test.add[1a638710-810a-422d-8f2f-d554af3c5a92]
[2017-05-27 13:46:22,538: INFO/MainProcess] Received task: celery_test.add[5b1a6863-1b62-4927-849b-e04e2d34ce7c]
[2017-05-27 13:46:22,540: INFO/MainProcess] Received task: celery_test.add[f4cd393a-2f02-48dd-b27f-d73806e154da]
[2017-05-27 13:46:22,543: INFO/MainProcess] Received task: celery_test.add[da8241bf-ce4e-4fe6-bd65-91350da8b163]
[2017-05-27 13:46:22,544: INFO/MainProcess] Received task: celery_test.add[4892b31a-e488-4011-86e7-d55eb941cf1f]
[2017-05-27 13:46:22,545: INFO/MainProcess] Received task: celery_test.add[c883b6ec-3842-4b50-bfab-35613f1724ed]
[2017-05-27 13:46:22,548: INFO/MainProcess] Received task: celery_test.add[1b021f4c-b41d-46a8-8548-7016539e8a8b]
[2017-05-27 13:46:22,549: INFO/MainProcess] Received task: celery_test.add[ae5d9b7d-0fa2-493b-aa0b-0b13b3764fdf]
[2017-05-27 13:46:22,551: INFO/MainProcess] Received task: celery_test.add[af0d27fe-394f-4fbb-821e-95acbb99324c]
[2017-05-27 13:46:22,552: INFO/MainProcess] Received task: celery_test.add[0b91cc9d-63c4-4a39-9d03-35ac474252bc]
[2017-05-27 13:46:22,553: INFO/MainProcess] Received task: celery_test.add[1be5a881-064c-4cf8-8a18-10c2fe0400ed]
[2017-05-27 13:46:24,538: INFO/PoolWorker-1] Task celery_test.add[402fb858-d9ec-4565-ab71-fbf4ec531787] succeeded in 2.0080740470439196s: 0
[2017-05-27 13:46:24,540: INFO/PoolWorker-3] Task celery_test.add[311fc7ed-e44a-4119-a0fa-c6849574723e] succeeded in 2.007692627608776s: 2
[2017-05-27 13:46:24,543: INFO/PoolWorker-6] Task celery_test.add[29234f7f-f841-44c2-94fb-b491a074318b] succeeded in 2.007782282307744s: 6
[2017-05-27 13:46:24,543: INFO/PoolWorker-5] Task celery_test.add[af54e423-651b-4b01-a3d1-26ead5ae6af1] succeeded in 2.007910629734397s: 4
[2017-05-27 13:46:24,546: INFO/PoolWorker-8] Task celery_test.add[1a638710-810a-422d-8f2f-d554af3c5a92] succeeded in 2.0075698774307966s: 8
[2017-05-27 13:46:24,548: INFO/PoolWorker-2] Task celery_test.add[f4cd393a-2f02-48dd-b27f-d73806e154da] succeeded in 2.0072230715304613s: 11
[2017-05-27 13:46:24,548: INFO/PoolWorker-10] Task celery_test.add[5b1a6863-1b62-4927-849b-e04e2d34ce7c] succeeded in 2.007256705313921s: 10
[2017-05-27 13:46:24,552: INFO/PoolWorker-7] Task celery_test.add[da8241bf-ce4e-4fe6-bd65-91350da8b163] succeeded in 2.0082139261066914s: 12
[2017-05-27 13:46:24,554: INFO/PoolWorker-9] Task celery_test.add[c883b6ec-3842-4b50-bfab-35613f1724ed] succeeded in 2.0077442210167646s: 14
[2017-05-27 13:46:24,554: INFO/PoolWorker-4] Task celery_test.add[4892b31a-e488-4011-86e7-d55eb941cf1f] succeeded in 2.007783567532897s: 13
[2017-05-27 13:46:26,542: INFO/PoolWorker-1] Task celery_test.add[1b021f4c-b41d-46a8-8548-7016539e8a8b] succeeded in 2.002950184047222s: 15
[2017-05-27 13:46:26,544: INFO/PoolWorker-3] Task celery_test.add[ae5d9b7d-0fa2-493b-aa0b-0b13b3764fdf] succeeded in 2.002891855314374s: 16
[2017-05-27 13:46:26,547: INFO/PoolWorker-5] Task celery_test.add[af0d27fe-394f-4fbb-821e-95acbb99324c] succeeded in 2.002899706363678s: 17
[2017-05-27 13:46:26,547: INFO/PoolWorker-6] Task celery_test.add[0b91cc9d-63c4-4a39-9d03-35ac474252bc] succeeded in 2.002899182960391s: 18
[2017-05-27 13:46:26,550: INFO/PoolWorker-8] Task celery_test.add[1be5a881-064c-4cf8-8a18-10c2fe0400ed] succeeded in 2.0029856264591217s: 19
但是，检索结果会延迟，总共需要 20 秒：
1.9911870956420898 0
0.0006098747253417969 1
0.0011210441589355469 2
2.003366231918335 3
0.0006439685821533203 4
2.0034918785095215 5
1.0012683868408203 6
1.00254487991333 7
1.001213788986206 8
1.002840518951416 9
1.0012362003326416 10
1.001204490661621 11
1.00126314163208 12
1.0012261867523193 13
1.0012249946594238 14
1.0012695789337158 15
1.0013458728790283 16
1.0013868808746338 17
1.0014445781707764 18
1.001399278640747 19
我有两个问题：
如何设置 celery 以优化工作分配？例如。一个好的解决方案是将 1+1 个任务分配给第一个节点，将 10+8 个任务分配给第二个节点。
恕我直言，检索结果的时间应该与最慢的工作人员一样长，即 10 秒。为什么需要更长的时间？如何加速？
关闭第一个worker，第二个worker 10+10个任务需要4秒（ok），取回结果需要5秒。为什么我还松了一秒？
Understanding celery task prefetching 将运行时间减少到 4 或 5 秒。还是有时候1秒太多，但是1.和2.都解决了。
由于这似乎是一个不同的问题，所以我在一个单独的问题中再次询问了 3.：