如何在 clojure 中为风暴螺栓设置 `setNumTasks`？

Question

我想为我的风暴拓扑中的螺栓设置任务数。但是我找不到在 clojure 中执行此操作的方法。我在 bolt-spec 的文档中也没有看到任何内容。我错过了什么吗？为了微调我的应用程序，我需要一种方法来设置任务数。这可能吗？

[1] : http://storm.apache.org/releases/0.10.0/Clojure-DSL.html

Answer 1

根据 Clojure-DSL 页面：

To create topology configs, it's easiest to use the backtype.storm.config namespace which defines constants for all of the possible configs. The constants are the same as the static constants in the Config class, except with dashes instead of underscores. For example, here's a topology config that sets the number of workers to 15 and configures the topology in debug mode:
{TOPOLOGY-DEBUG true
 TOPOLOGY-WORKERS 15}

并且来自 Storm 文档（查找名为 "Number of tasks" 的部分）：

Number of tasks

Description: How many tasks to create per component.

Configuration option: TOPOLOGY_TASKS

How to set in your code (examples):

ComponentConfigurationDeclarer#setNumTasks()

Here is an example code snippet to show these settings in practice:
topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)
               .setNumTasks(4)
               .shuffleGrouping("blue-spout");

看起来使用 backtype.storm.config 命名空间并定义 TOPOLOGY-WORKERS 等同于调用 setNumTasks.

Answer 2

我相信你可以在调用 bolt spec 时在配置图中传递拓扑任务的数量

(bolt-spec {"1" :shuffle} geocode-lookup :p 8 :conf {TOPOLOGY_TASKS 64})

这是一个示例拓扑：

(defn heatmap-topology []
  (topology
   {"1" (spout-spec checkins :p 4)}

   {"2" (bolt-spec {"1" :shuffle} geocode-lookup :p 8 :conf {TOPOLOGY_TASKS 64})
    "3" (bolt-spec {"2" :shuffle} time-interval-extractor :p 4)
    "4" (bolt-spec {"3" ["time-interval" "city"]} heatmap-builder :p 4)
    "5" (bolt-spec {"4" :shuffle} persistor :conf {TOPOLOGY_TASKS 4})}))

我也在想同样的事情，发现了这个：

org.apache.storm.clojure/bolt-spec 是 org.apache.storm.thrift/mk-bolt-spec link
bolt-spec 是使用 org.apache.storm.util/defnk link 定义的，所以它 returns 像 {:obj bolt :inputs inputs :p parallelism-hint :conf conf} 这样的映射，conf 默认为 nil
org.apache.storm.clojure/bolt-spec 是 org.apache.storm.thrift/mk-topology 的别名，调用 org.apache.storm.topology.TopologyBuilder.addConfigurations 传递 conf 作为唯一参数。 link
此外，setNumTasks 简单调用 addConfiguration(Config.TOPOLOGY_TASKS, val)link
最后，设置一个 bolt 任务数的正确常量是 TOPOLOGY_TASKS link

如何在 clojure 中为风暴螺栓设置 `setNumTasks`？

How to set `setNumTasks` for a storm bolt in clojure?

clojure

apache-storm

Number of tasks