根据字符串列表中不存在的列值过滤行

Question

我有一个数据框

var input1 = spark.createDataFrame(Seq(
        (10L, "Joe Doe", 34),
        (11L, "Jane Doe", 31),
        (12L, "Alice Jones", 25)
        )).toDF("id", "name", "age")

我正在尝试过滤列表中不可用的行。我可以轻松地根据年龄和身份进行过滤 -

input1.filter("age not in (31,56,81)").show()

但是当我尝试根据名称进行过滤时同样不起作用

input1.filter("name not in ("joe Doe","Pappu cam","Log")").show()

过滤时必须有一些字符串表示。

我收到异常

org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input 'Doe' expecting {')', ',', '.', '[', 'OR', 'AND', 'IN', NOT, 'BETWEEN', 'LIKE', RLIKE, 'IS', EQ, '<=>', '<>', '!=', '<', LTE, '>', GTE, '+', '-', '*', '/', '%', 'DIV', '&', '|', '^'}(line 1, pos 16)
== SQL ==
name not in (Joe Doe,abc dej)
----------------^^^

Answer 1

似乎是语法错误。尝试：

input1.filter("name not in ('joe Doe','Pappu cam','Log')").show()

Answer 2

尝试转义 SQL 查询：

input1.filter(s"""name not in ("joe Doe","Pappu cam","Log")""").show()

根据字符串列表中不存在的列值过滤行

Filter row based on column value not present in List Of String or not

apache-spark

spark-dataframe