You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Valentyn Tymofieiev (JIRA)" <ji...@apache.org> on 2018/03/09 22:12:00 UTC
[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for
2.4.0 RC2
[ https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393681#comment-16393681 ]
Valentyn Tymofieiev commented on BEAM-3824:
-------------------------------------------
Also do we understand why the error did not surface earlier?
> BQ sink fails on Direct Runner for 2.4.0 RC2
> --------------------------------------------
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Reporter: Valentyn Tymofieiev
> Assignee: Chamikara Jayalath
> Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score --project=$PROJECT --dataset=beam_release_2_4_0 --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish <DoOperation WriteTeamScoreSums/WriteToBigQuery output_tags=['out'], receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]>
> Traceback (most recent call last):
> File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", line 276, in <
> module>
> run()
> File "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py", line 270, in r
> un
> write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND))
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 389, in __exit__
> self.run().wait_until_finish()
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 369, in run
> self.to_runner_api(), self.runner, self._options).run(False)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 382, in run
> return self.runner.run_pipeline(self)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 129, in run_pip
> eline
> return runner.run_pipeline(pipeline)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 215, in ru
> n_pipeline
> return self.run_via_runner_api(pipeline.to_runner_api())
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 218, in ru
> n_via_runner_api
> return self.run_stages(*self.create_stages(pipeline_proto))
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 837, in ru
> n_stages
> pcoll_buffers, safe_coders).process_bundle.metrics
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 938, in ru
> n_stage
> self._progress_frequency).process_bundle(data_input, data_output)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 1110, in p
> rocess_bundle
> result_future = self._controller.control_handler.push(process_bundle)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 1003, in p
> ush
> response = self.worker.do_instruction(request)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 185, in do_instruc
> tion
> request.instruction_id)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 202, in process_bu
> ndle
> processor.process_bundle(instruction_id)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 298, in proc
> ess_bundle
> op.finish()
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py", line 389, in finish
> self.dofn_runner.finish()
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 517, in finish
> self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 508, in _invoke_bundle_method
> self._reraise_augmented(exn)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py", line 537, in _reraise_augmented
> six.raise_from(new_exn, original_traceback)
> File "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/six.py", line 737, in raise_from
> raise value
> RuntimeError: Could not successfully insert rows to BigQuery table [google.com:clouddfe:beam_release_2_4_0.leader_board]. Errors: [<InsertEr
> rorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> index: 0>, <InsertErrorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> index: 1>, <InsertErrorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> index: 2>, <InsertErrorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> index: 3>, <InsertErrorsValueListEntry
> errors: [<ErrorProto
> debugInfo: u''
> location: u'processing_time'
> message: u'no such field.'
> reason: u'invalid'>]
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)