用于结构化流的 Elasticsearch 火花连接

Question

我正在尝试从我的 spark 程序连接到 elasticsearch。我的 elasticsearch 主机是 https，但没有找到连接属性。我们正在使用 spark structured streaming Java API 连接细节如下，

        SparkSession spark = SparkSession.builder()
                .config(ConfigurationOptions.ES_NET_HTTP_AUTH_USER, "username")
                .config(ConfigurationOptions.ES_NET_HTTP_AUTH_PASS, "password")
                 .config(ConfigurationOptions.ES_NODES, "my_host_url")
                 .config(ConfigurationOptions.ES_PORT, "9200")
.config(ConfigurationOptions.ES_NET_SSL_TRUST_STORE_LOCATION,"C:\certs\elastic\truststore.jks")
.config(ConfigurationOptions.ES_NET_SSL_TRUST_STORE_PASS,"my_password") .config(ConfigurationOptions.ES_NET_SSL_KEYSTORE_TYPE,"jks")
                .master("local[2]")
                .appName("spark_elastic").getOrCreate();
        spark.conf().set("spark.sql.shuffle.partitions",2);
        spark.conf().set("spark.default.parallelism",2);

我收到以下错误

19/07/01 12:26:00 INFO HttpMethodDirector: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server 10.xx.xxx.xxx failed to respond
19/07/01 12:26:00 INFO HttpMethodDirector: Retrying request
19/07/01 12:26:00 ERROR NetworkClient: Node [10.xx.xxx.xxx:9200] failed (The server 10.xx.xxx.xxx failed to respond); no other nodes left - aborting...
19/07/01 12:26:00 ERROR StpMain: Error
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
    at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:344)

可能是因为它试图通过 http 协议启动连接，但在我的情况下，我需要 https 连接，但不确定如何配置

Answer 1

错误发生是因为 spark 无法找到信任库文件。看来我们需要为要接受的路径添加"file:\"。

用于结构化流的 Elasticsearch 火花连接

Elasticsearch spark connection for structured-streaming

elasticsearch

apache-spark

spark-structured-streaming