Beam mongoio DisplayDataItem Value {} 是不受支持的类型

Question

当我尝试通过

将文档写入mongodb时

         | "Write User Doc to Mongo" >> beam.io.WriteToMongoDB(uri=MONGO_URI,
                                                               db="dbname",
                                                               coll="col_name"
                                                               ))

错误ValueError: Invalid DisplayDataItem. Value {} is of an unsupported type.

ERROR:root:Error while visiting Write User Doc to Mongo/ParDo(_WriteMongoFn)
Traceback (most recent call last):
  File "beam_home.py", line 317, in <module>
    run()
  File "beam_home.py", line 312, in run
    p.run().wait_until_finish()
  File "/usr/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 406, in run
    self._options).run(False)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 419, in run
    return self.runner.run_pipeline(self, self._options)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 469, in run_pipeline
    super(DataflowRunner, self).run_pipeline(pipeline, options)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/runner.py", line 158, in run_pipeline
    pipeline.visit(RunVisitor(self))
  File "/usr/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 447, in visit
    self._root_transform().visit(visitor, self, visited)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 824, in visit
    part.visit(visitor, pipeline, visited)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 824, in visit
    part.visit(visitor, pipeline, visited)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 827, in visit
    visitor.visit_transform(self)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/runner.py", line 153, in visit_transform
    self.runner.run_transform(transform_node, options)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/runner.py", line 196, in run_transform
    return m(transform_node, options)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 807, in run_ParDo
    transform_node.transform.output_tags)
  File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 590, in _add_step
    DisplayData.create_from(transform_node.transform).items])
  File "/usr/local/lib/python2.7/site-packages/apache_beam/transforms/display.py", line 274, in get_dict
    self.is_valid()
  File "/usr/local/lib/python2.7/site-packages/apache_beam/transforms/display.py", line 246, in is_valid
    .format(self.value))
ValueError: Invalid DisplayDataItem. Value {} is of an unsupported type.

好像是这个引起的codes

    if extra_client_params is None:
      extra_client_params = {}
    if not isinstance(db, str):

看来我必须设置 extra_client_params of WriteToMongoDB。这是默认行为还是我遗漏了什么？

Answer 1

不幸的是，这是一个错误。将在 2.16.0 中对其进行修复，应该会在接下来的几周内发布。

主要问题就在这里：https://github.com/apache/beam/blob/v2.15.0/sdks/python/apache_beam/io/mongodbio.py#L472

显示数据不支持字典类型的参数，我们正在通过规范。要解决该错误，不幸的是，您必须在本地 Beam 代码中覆盖该行为 :( - 或等待几周 2.16.0 :)

Beam mongoio DisplayDataItem Value {} 是不受支持的类型

Beam mongoio DisplayDataItem Value {} is of an unsupported type

python

google-cloud-dataflow

apache-beam