pyspark datastax cassandra 连接器保持连接到本地主机

Question

我正在尝试使用 datastax 驱动程序将 pyspark 连接到 cassandra

 conf = SparkConf()\
.setAppName('Test') \
.setMaster('local[4]') \
.set("spark.cassandra.connection.host", "192.168.0.150") 
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)

df = sqlContext.read.format("org.apache.spark.sql.cassandra").\
options(table="test", keyspace="test_keyspace").load()

出于某种原因，它一直连接到 127.0.0.1:9042 而不是 192.168.0.150

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All  
host(s) tried for query 
failed (tried: localhost/127.0.0.1:9042  
(com.datastax.driver.core.exceptions.TransportException: 
[localhost/127.0.0.1] Cannot connect))

我正在使用 spark 2.10 和运行程序如下

spark-submit --packages datastax:spark-cassan
dra-connector:2.0.0-RC1-s_2.11 test.py

Answer 1

找出 Spark 2.10 中的问题 cassandra 配置被设置为 SqlContext 中的选项。下面的代码有效

sqlContext.read.format("org.apache.spark.sql.cassandra").\
option("spark.cassandra.connection.host", "192.168.0.150").\
option("spark.cassandra.auth.username", "user"). \
option("spark.cassandra.auth.password", "password"). \
options(table="test_table", keyspace="test_space").load()

pyspark datastax cassandra 连接器保持连接到本地主机

pyspark datastax cassandra connector keeps connecting to localhost

python

datastax

pyspark

spark-cassandra-connector

pyspark-sql