无法在 jupyter notebook 中使用 pyspark 读取 MS SQL table？

Question

import os
import sys

spark_path = 'C:/opt/spark/spark-2.4.1-bin-hadoop2.7'

os.environ['SPARK_HOME'] = spark_path
os.environ['HADOOP_HOME'] = spark_path

sys.path.append(spark_path + "/bin")
sys.path.append(spark_path + "/python")
sys.path.append(spark_path + "/python/pyspark/")
sys.path.append(spark_path + "/python/lib")
sys.path.append(spark_path + "/python/lib/pyspark.zip")
sys.path.append(spark_path + "/python/lib/py4j-0.9-src.zip")

from pyspark.sql import SparkSession
spark = SparkSession\
    .builder\
    .master('local[*]')\
    .appName('Connection-Test')\
    .config('spark.driver.extraClassPath', 'C:/Users/sqljdbc_4.2.8112.200_enu/sqljdbc_4.2/enu/jre8/sqljdbc42.jar')\
    .config('spark.executor.extraClassPath', 'C:/Users/sqljdbc_4.2.8112.200_enu/sqljdbc_4.2/enu/jre8/sqljdbc42.jar')\
    .getOrCreate()


sqlsUrl = 'jdbc:sqlserver://ip:port;database=dbname'

qryStr = """ (
    SELECT *
    FROM Table
    )  """

spark.read.format('jdbc')\
    .option('url',sqlsUrl)\
    .option('driver', 'com.microsoft.sqlserver.jdbc.SQLServerDriver')\
    .option('dbtable', qryStr )\
    .option("user", "user") \
    .option("password", "password") \
    .load().show()

调用 o50.load 时出错。 : com.microsoft.sqlserver.jdbc.SQLServerException: 关键字 'WHERE' 附近的语法不正确。在 com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)

Answer 1

尝试在查询的末尾添加 "as Table_Name"

qryStr = """ (
    SELECT *
    FROM Table
    ) as Table """

无法在 jupyter notebook 中使用 pyspark 读取 MS SQL table？

Unable to read MS SQL table using pyspark in jupyter notebook?

python-3.x

pyspark-sql

jupyter-notebook