3
我有一個看似簡單的場景,我正在使用python數據流來查詢使用big-query的數據。通過python中的bigquery閱讀器讀取行時AssertError谷歌雲數據流
當bq查詢返回零行時,我遇到一個AssertionError,腳本&聲明錯誤如下所示。我想知道如果這是一個錯誤,或者可能有一個推薦的方式來處理py數據流中的bq閱讀器零行?
數據流腳本:
from apache_beam.io import WriteToText
from apache_beam.typehints import Any, Dict
pipeline_options = PipelineOptions(pipeline_args)
pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
BIGQUERY_ROW_TYPE = Dict[str, Any]
# construct a bigquery SQL
query_sql = Query().build_sql()
lines = p \
| 'read from bigquery' >> beam.io.Read(beam.io.BigQuerySource(query=query_sql, validate=True)).with_output_types(BIGQUERY_ROW_TYPE) \
| 'write to test' >> WriteToText(known_args.output)
result = p.run()
,我看到當查詢返回零行錯誤:
(98b5a6e4c0cd002e): Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 581, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 166, in execute
op.start()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/native_operations.py", line 48, in start
for value in reader:
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/nativefileio.py", line 186, in __iter__
for eof, record, delta_offset in self.read_records():
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/nativeavroio.py", line 102, in read_records
assert block.num_records() > 0
AssertionError
`2017-06-27 (13:55:58) Workflow failed. Causes: (7390b72dc5ceedb6): S04:read from bigquery+write to test/Write/WriteImpl/Wr...
(bb74ab934e658b06): Workflow failed. Causes: (7390b72dc5ceedb6): S04:read
from bigquery+write to test/Write/WriteImpl/WriteBundles/Do+write to
test/Write/WriteImpl/Pair+write to
test/Write/WriteImpl/WindowInto(WindowIntoFn)+write to
test/Write/WriteImpl/GroupByKey/Reify+write to
test/Write/WriteImpl/GroupByKey/Write failed.`
你有一個實際的查詢嗎? – Pablo
我相信這是Python Dataflow worker中的一個bug。 Avro規範不要求塊具有非零元素計數,但工作人員不正確地強制執行此要求。我們正在修復。 – jkff
@jkff關於這個的任何消息? – Goranek