[jira] [Commented] (BEAM-8965) WriteToBigQuery failed in BundleBasedDirectRunner

["Wenbing Bai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995849#comment-16995849 ] 

Wenbing Bai commented on BEAM-8965:
-----------------------------------

I did a small investigate, I think Singleton object  is somehow evaluated twice BundleBasedDirectRunner.

Here is the Singleton object [https://github.com/apache/beam/blob/de30361359b70e9fe9729f0f3d52f6c6e8462cfb/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L802]

And the Singleton object is used twice in [https://github.com/apache/beam/blob/de30361359b70e9fe9729f0f3d52f6c6e8462cfb/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L656]

*Also note*, this is only for BundleBasedDirectRunner, good at DataFlowRunner and FnApiRunner.

 

> WriteToBigQuery failed in BundleBasedDirectRunner
> -------------------------------------------------
>
>                 Key: BEAM-8965
>                 URL: https://issues.apache.org/jira/browse/BEAM-8965
>             Project: Beam
>          Issue Type: Bug
>          Components: io-py-gcp
>    Affects Versions: 2.16.0
>            Reporter: Wenbing Bai
>            Priority: Major
>
> {{*{{WriteToBigQuery}}* failed in }}{{*BundleBasedDirectRunner*}}{{ with error PCollection of size 2 with more than one element accessed as a singleton view.}}
> Here is the code
>  
> {code:java}
> with Pipeline() as p:
>     query_results = (
>         p 
>         | beam.io.Read(beam.io.BigQuerySource(
>             query='SELECT ... FROM ...')
>         )
>     query_results | beam.io.gcp.WriteToBigQuery(
>             table=<your_table_name>,
>             method=WriteToBigQuery.Method.FILE_LOADS,
>             schema={"fields": []}
>         )
> {code}
>  
> Here is the error
>  
> {code:java}
>   File "apache_beam/runners/common.py", line 778, in apache_beam.runners.common.DoFnRunner.process
>     def process(self, windowed_value):
>   File "apache_beam/runners/common.py", line 782, in apache_beam.runners.common.DoFnRunner.process
>     self._reraise_augmented(exn)
>   File "apache_beam/runners/common.py", line 849, in apache_beam.runners.common.DoFnRunner._reraise_augmented
>     raise_with_traceback(new_exn)
>   File "apache_beam/runners/common.py", line 780, in apache_beam.runners.common.DoFnRunner.process
>     return self.do_fn_invoker.invoke_process(windowed_value)
>   File "apache_beam/runners/common.py", line 587, in apache_beam.runners.common.PerWindowInvoker.invoke_process
>     self._invoke_process_per_window(
>   File "apache_beam/runners/common.py", line 610, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
>     [si[global_window] for si in self.side_inputs]))
>   File "/home/wbai/terra/terra_py2/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__
>     _FilteringIterable(self._iterable, target_window), self._view_options)
>   File "/home/wbai/terra/terra_py2/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 443, in _from_runtime_iterable
>     len(head), str(head[0]), str(head[1])))
> ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "gs://temp-dev/temp/bq_load/3edbf2172dd540edb5c8e9597206b10f", "gs://temp-dev/temp/bq_load/3edbf2172dd540edb5c8e9597206b10f". [while running 'WriteToBigQuery/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)']
> {code}
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)