[jira] [Updated] (BEAM-7471) DirectRunner GameStats fails, ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']

["Ismaël Mejía (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ismaël Mejía updated BEAM-7471:
-------------------------------
    Priority: Major  (was: Blocker)

> DirectRunner GameStats fails, ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-7471
>                 URL: https://issues.apache.org/jira/browse/BEAM-7471
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-gearpump
>    Affects Versions: 2.13.0
>            Reporter: Ankur Goenka
>            Assignee: Ankur Goenka
>            Priority: Major
>
>  
> GameStats example is failing with following logs 
> {noformat}
> *****************************************************
> * Running GameStats with DirectRunner
> *****************************************************
> INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
> /usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1137: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
> method_to_use = self._compute_method(p, p.options)
> INFO:root:Running pipeline with DirectRunner.
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> INFO:root:Created table google.com:clouddfe.goenka_python_validations.game_stats_teams with schema <TableSchema
> fields: [<TableFieldSchema
> fields: []
> mode: u'NULLABLE'
> name: u'processing_time'
> type: u'STRING'>, <TableFieldSchema
> fields: []
> mode: u'NULLABLE'
> name: u'total_score'
> type: u'INTEGER'>, <TableFieldSchema
> fields: []
> mode: u'NULLABLE'
> name: u'window_start'
> type: u'STRING'>, <TableFieldSchema
> fields: []
> mode: u'NULLABLE'
> name: u'team'
> type: u'STRING'>]>. Result: <Table
> creationTime: 1559341252301
> etag: u'tEQeN0Sr4FvttBRZuXn7zw=='
> id: u'google.com:clouddfe:goenka_python_validations.game_stats_teams'
> kind: u'bigquery#table'
> lastModifiedTime: 1559341252336
> location: u'US'
> numBytes: 0
> numLongTermBytes: 0
> numRows: 0
> schema: <TableSchema
> fields: [<TableFieldSchema
> fields: []
> mode: u'NULLABLE'
> name: u'processing_time'
> type: u'STRING'>, <TableFieldSchema
> fields: []
> mode: u'NULLABLE'
> name: u'total_score'
> type: u'INTEGER'>, <TableFieldSchema
> fields: []
> mode: u'NULLABLE'
> name: u'window_start'
> type: u'STRING'>, <TableFieldSchema
> fields: []
> mode: u'NULLABLE'
> name: u'team'
> type: u'STRING'>]>
> selfLink: u'https://www.googleapis.com/bigquery/v2/projects/google.com:clouddfe/datasets/goenka_python_validations/tables/game_stats_teams'
> tableReference: <TableReference
> datasetId: u'goenka_python_validations'
> projectId: u'google.com:clouddfe'
> tableId: u'game_stats_teams'>
> type: u'TABLE'>.
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception.
> Traceback (most recent call last):
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call
> finish_state)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call
> evaluator.process_element(value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element
> self.runner.process(element)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process
> self._reraise_augmented(exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented
> raise_with_traceback(new_exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window
> side_inputs = [si[window] for si in self.side_inputs]
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__
> _FilteringIterable(self._iterable, target_window), self._view_options)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable
> return options['data'].view_fn(it)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda>
> lambda iterable: from_runtime_iterable(iterable, view_options))
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable
> len(head), str(head[0]), str(head[1])))
> ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
> ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception.
> Traceback (most recent call last):
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call
> finish_state)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call
> evaluator.process_element(value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element
> self.runner.process(element)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process
> self._reraise_augmented(exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented
> raise_with_traceback(new_exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window
> side_inputs = [si[window] for si in self.side_inputs]
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__
> _FilteringIterable(self._iterable, target_window), self._view_options)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable
> return options['data'].view_fn(it)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda>
> lambda iterable: from_runtime_iterable(iterable, view_options))
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable
> len(head), str(head[0]), str(head[1])))
> ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
> ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception.
> Traceback (most recent call last):
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call
> finish_state)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call
> evaluator.process_element(value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element
> self.runner.process(element)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process
> self._reraise_augmented(exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented
> raise_with_traceback(new_exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window
> side_inputs = [si[window] for si in self.side_inputs]
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__
> _FilteringIterable(self._iterable, target_window), self._view_options)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable
> return options['data'].view_fn(it)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda>
> lambda iterable: from_runtime_iterable(iterable, view_options))
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable
> len(head), str(head[0]), str(head[1])))
> ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
> ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception.
> Traceback (most recent call last):
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call
> finish_state)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call
> evaluator.process_element(value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element
> self.runner.process(element)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process
> self._reraise_augmented(exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented
> raise_with_traceback(new_exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window
> side_inputs = [si[window] for si in self.side_inputs]
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__
> _FilteringIterable(self._iterable, target_window), self._view_options)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable
> return options['data'].view_fn(it)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda>
> lambda iterable: from_runtime_iterable(iterable, view_options))
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable
> len(head), str(head[0]), str(head[1])))
> ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
> ERROR:root:Giving up after 4 attempts.
> WARNING:root:A task failed with exception: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
> INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
> INFO:oauth2client.client:Refreshing access_token
> Traceback (most recent call last):
> File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py", line 399, in <module>
> run()
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py", line 393, in run
> }, options.view_as(GoogleCloudOptions).project))
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 426, in __exit__
> self.run().wait_until_finish()
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 430, in wait_until_finish
> self._executor.await_completion()
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 400, in await_completion
> self._executor.await_completion()
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 446, in await_completion
> raise_(t, v, tb)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 343, in call
> finish_state)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 380, in attempt_call
> evaluator.process_element(value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 633, in process_element
> self.runner.process(element)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 753, in process
> self._reraise_augmented(exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 807, in _reraise_augmented
> raise_with_traceback(new_exn)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 751, in process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 564, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 571, in _invoke_process_per_window
> side_inputs = [si[window] for si in self.side_inputs]
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py", line 65, in __getitem__
> _FilteringIterable(self._iterable, target_window), self._view_options)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 346, in _from_runtime_iterable
> return options['data'].view_fn(it)
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 323, in <lambda>
> lambda iterable: from_runtime_iterable(iterable, view_options))
> File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py", line 433, in _from_runtime_iterable
> len(head), str(head[0]), str(head[1])))
> ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)