Flink：HA模式杀死领先的jobmanager终止备用jobmanagers

Question

我正在尝试使用 Zookeeper 在 HA 模式下使 Flink 达到运行，但是当我尝试通过杀死领导者 JobManager 来测试它时，我所有的备用 jobmanagers 也被杀死了。

因此，不是一个备用的 jobmanager 接管了新的 Leader，而是他们都被杀死了，这是不应该发生的。

我的设置： 4 台服务器，其中 3 台服务器有 Zookeeper 运行ning，但只有 1 台服务器将托管所有 JobManager。

ad011.local: Zookeeper + Jobmanagers
ad012.local: Zookeeper + Taskmanager
ad013.local: Zookeeper
ad014.local: nothing interesting

我的主文件如下所示：

ad011.local:8081
ad011.local:8082
ad011.local:8083

我的flink-conf.yaml:

jobmanager.rpc.address: ad011.local

blob.server.port: 6130,6131,6132

jobmanager.heap.mb: 512
taskmanager.heap.mb: 128
taskmanager.numberOfTaskSlots: 4
parallelism.default: 2
taskmanager.tmp.dirs: /var/flink/data

metrics.reporters: jmx
metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.jmx.port: 8789,8790,8791

high-availability: zookeeper
high-availability.zookeeper.quorum: ad011.local:2181,ad012.local:2181,ad013.local:2181

high-availability.zookeeper.path.root: /flink
high-availability.zookeeper.path.cluster-id: /cluster-one
high-availability.storageDir: /var/flink/recovery
high-availability.jobmanager.port: 50000,50001,50002

当我使用 start-cluster.sh 脚本运行 flink 时，我看到我的 3 个 JobManager 运行ning，然后转到 WebUI，它们都指向 ad011.local： 8081，这是领导者。我猜哪个可以？

然后我尝试通过使用 kill 杀死领导者来测试故障转移，然后我所有其他备用 JobManager 也停止。

这是我在备用 JobManager 日志中看到的内容：

2017-09-29 08:08:41,590 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager at akka.tcp://flink@ad011.local:50002/user/jobmanager.
2017-09-29 08:08:41,590 INFO  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService  - Starting ZooKeeperLeaderElectionService org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService@72d546c8.
2017-09-29 08:08:41,598 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting with JobManager akka.tcp://flink@ad011.local:50002/user/jobmanager on port 8083
2017-09-29 08:08:41,598 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting ZooKeeperLeaderRetrievalService.
2017-09-29 08:08:41,645 INFO  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader reachable under akka.tcp://flink@ad011.local:50000/user/jobmanager:f7dc2c48-dfa5-45a4-a63e-ff27be21363a.
2017-09-29 08:08:41,651 INFO  org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting ZooKeeperLeaderRetrievalService.
2017-09-29 08:08:41,722 INFO  org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager  - Received leader address but not running in leader ActorSystem. Cancelling registration.
2017-09-29 09:26:13,472 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@ad011.local:50000] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
2017-09-29 09:26:14,274 INFO  org.apache.flink.runtime.jobmanager.JobManager                - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2017-09-29 09:26:14,284 INFO  org.apache.flink.runtime.blob.BlobServer                      - Stopped BLOB server at 0.0.0.0:6132

如有任何帮助，我们将不胜感激。

Answer 1

通过运行我的集群使用 ./bin/start-cluster.sh 而不是使用服务文件（调用相同的脚本）解决了这个问题，服务文件显然杀死了其他作业管理器。

Flink：HA模式杀死领先的jobmanager终止备用jobmanagers

Flink: HA mode killing leading jobmanager terminating standby jobmanagers

apache-flink

apache-zookeeper