pyspark error: not enough values to unpack(expected 3, got 2)

Question

我有以下数据

my_data=["Sentence1",['a','b','c'],"Sentence2",['d','e'],"Sentence3",['f']]

并希望使用 pyspark 2.2 as

将输入数据转换为数据帧

---------------------------------
| ID | Sentence  | label        |
---------------------------------
| 01 | Sentence1 |['a','b','c'] |
| 02 | Sentence2 |['d','e']     |

但是当使用下面的代码

from pyspark.sql import Row
to_row = Row('ID','Sentence', 'list')
df = spark.createDataFrame([to_row(i,s,l) for i,s,l in enumerate(my_data)])

我收到以下错误

ValueError: not enough values to unpack (expected 3, got 2)

有人可以帮助我吗？

Answer 1

您似乎复制并粘贴了所以：

enumerate:
yields pairs containing a count (from start, which defaults to zero) and a value yielded by the iterable argument. enumerate is useful for obtaining an indexed list:
```
(0, seq[0]), (1, seq[1]), (2, seq[2]), ...
```
使用 mydata 它将产生
```
[(0, 'Sentence1'),
 (1, ['a', 'b', 'c']),
 (2, 'Sentence2'),
 (3, ['d', 'e']),
 (4, 'Sentence3'),
 (5, ['f'])]
```
这根本没用。

这是因为你的数据不应该

["Sentence1",['a','b','c'],"Sentence2",['d','e'],"Sentence3",['f']]

但是

[("Sentence1",['a','b','c']),("Sentence2",['d','e']),("Sentence3",['f'])]

如果要连接记录。

如果是这种形式，解包用
```
i,s,l 
```
不会工作，因为模式与数据不匹配。匹配模式为
```
i, (s, l)
```

合并：

from itertools import islice

spark.createDataFrame([(x, y, z) for x, (y, z) in enumerate(zip(
    islice(my_data, 0, len(my_data), 2),
    islice(my_data, 1, len(my_data), 2)))
], ("id", "sentence", "label"))

哪里

islice(my_data, m, len(my_data), n)

从第 m 个开始从列表中取出第 n 个项目，zip(seq1, seq2) 将序列合并为：

[(seq1[0], seq2[0]), (seq1[1], seq2[1]), ... ]

和 enumerate 将其转换为

[(0, (seq1[0], seq2[0])), (1, (seq1[1], seq2[1])), ... ]

在切换到更高级的工具之前，请考虑学习这门语言。

pyspark error: not enough values to unpack(expected 3, got 2)

pyspark error: not enough values to unpack(expected 3, got 2)

python

apache-spark

pyspark

pyspark-sql