Elk Logstash 无法连接到事件中心 Azure

Question

我正在尝试从 ElasticSearch logstash 连接到 Azure 事件中心。这是我在 /etc/logstash/logstash.yml 文件中提到的 azure 模块的详细信息。

modules:
  - name: azure
    var.elasticsearch.hosts: localhost:9200
    var.kibana.host: localhost:5601
    var.input.azure_event_hubs.consumer_group: "default"
    var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=ABC;AccountKey=XXXXX==;EndpointSuffix=core.windows.net"
    var.input.azure_event_hubs.threads: 9
    var.input.azure_event_hubs.event_hub_connections:
      - "Endpoint=sb://elk-event.servicebus.windows.net/;SharedAccessKeyName=share-log;SharedAccessKey=YYYYY=;EntityPath=elk-log-event"

但是当我使用命令启动 logstash 时：

./logstash --path.settings=/etc/logstash

我使用 consumer_group 作为 logstash 以及默认值。有问题吗？

它抛出错误为：

[2019-10-04T09:43:46,332][ERROR][com.microsoft.azure.eventprocessorhost.PartitionPump] host logstash-8331671e-0dea-4f8d-88de-ba3ebc974dca: 1: PartitionReceiver creation failed
java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'elk-event:eventhub:elk-log-event~32766|default' could not be found. TrackingId:c380d842-e946-4f12-ba12-daecdf60df2d_B29, SystemTracker:elk-event:eventhub:elk-log-event~32766|default, Timestamp:2019-10-04T09:43:45 TrackingId:778a3beb42734bc098c7419f5f4b8c21_G5, SystemTracker:gateway5, Timestamp:2019-10-04T09:43:45, errorContext[NS: elk-event.servicebus.windows.net, PATH: elk-log-event/ConsumerGroups/default/Partitions/1, REFERENCE_ID: e393e0_8c21_G5_1570182225737]
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_222]
        at com.microsoft.azure.eventhubs.impl.ExceptionUtil.completeExceptionally(ExceptionUtil.java:98) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.cancelOpen(MessageReceiver.java:357) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.onOpenComplete(MessageReceiver.java:347) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.onError(MessageReceiver.java:414) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:736) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:74) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:113) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48) ~[azure-eventhubs-2.2.0.jar:?]
        at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) ~[proton-j-0.33.0.jar:?]
        at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) ~[proton-j-0.33.0.jar:?]
        at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) ~[proton-j-0.33.0.jar:?]
        at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) ~[proton-j-0.33.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:507) ~[azure-eventhubs-2.2.0.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access1(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'elk-event:eventhub:elk-log-event~32766|default' could not be found. TrackingId:c380d842-e946-4f12-ba12-daecdf60df2d_B29, SystemTracker:elk-event:eventhub:elk-log-event~32766|default, Timestamp:2019-10-04T09:43:45 TrackingId:778a3beb42734bc098c7419f5f4b8c21_G5, SystemTracker:gateway5, Timestamp:2019-10-04T09:43:45, errorContext[NS: elk-event.servicebus.windows.net, PATH: elk-log-event/ConsumerGroups/default/Partitions/1, REFERENCE_ID: e393e0_8c21_G5_1570182225737]
        at com.microsoft.azure.eventhubs.impl.ExceptionUtil.distinguishNotFound(ExceptionUtil.java:86) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:31) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:735) ~[azure-eventhubs-2.2.0.jar:?]
        ... 15 more

是跟消费群体有关还是其他？如果是这样，我不确定如何创建自己的消费者组。

有什么反馈吗？提前致谢。

Answer 1

消费者组的名称需要是$Default

//编辑：我应该注意，通常不建议使用 $Default 消费者组。最佳做法是为每个从事件中心读取数据的应用程序创建一个新的使用者组——即使只有一个应用程序。您永远不知道什么时候有人可能有其他想法并且也想读取数据...另外，命名消费者组可以让您轻松了解哪些应用正在从您的中心读取数据。

Elk Logstash 无法连接到事件中心 Azure

Elk Logstash not able to connect to Event Hub Azure

azure

elasticsearch

logstash

azure-eventhub