[jira] [Comment Edited] (BEAM-14016) apache_beam.io.gcp.experimental.spannerio_write_it_test.SpannerWriteIntegrationTest.test_spanner_update fails in Python3.7 Postcommits

["Ning (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500992#comment-17500992 ] 

Ning edited comment on BEAM-14016 at 3/3/22, 8:11 PM:
------------------------------------------------------

Just looking at the code, the [self.service_metric.call('ok')|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/experimental/spannerio.py#L1251] is misplaced:
* If all elements within the bundle are indeed handled by a spanner batched write mutation to the same table, the metric should be built only once when the bundle is not empty
* If each element needs to build their own metric, the "call" should be invoked inside the for-loop for each element in the bundle.

The only feasible execution route to cause the error is when the bundle is empty.
And I'm going to make sure that the metric is built at most once in each bundle and invoke the "call" only when the bundle is not empty.


was (Author: ningk):
Just looking at the code, the [self.service_metric.call('ok')|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/experimental/spannerio.py#L1251] is misplaced:
* If all elements within the bundle are indeed handled by a spanner batched write mutation to the same table, the metric should be built only once when the bundle is not empty
* If each element needs to build their own metric, the "call" should be invoked inside the for-loop for each element in the bundle.

The only feasible execution route to cause the error is when the bundle is empty.


>  apache_beam.io.gcp.experimental.spannerio_write_it_test.SpannerWriteIntegrationTest.test_spanner_update fails in Python3.7 Postcommits 
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-14016
>                 URL: https://issues.apache.org/jira/browse/BEAM-14016
>             Project: Beam
>          Issue Type: Bug
>          Components: io-py-gcp
>            Reporter: Valentyn Tymofieiev
>            Assignee: Ning
>            Priority: P1
>
> For some reason only Python 3.7 suite is failing with this error.
> {noformat}
> >             self)
> E         apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
> E         Traceback (most recent call last):
> E           File "apache_beam/runners/common.py", line 1198, in apache_beam.runners.common.DoFnRunner.process
> E           File "apache_beam/runners/common.py", line 537, in apache_beam.runners.common.SimpleInvoker.invoke_process
> E           File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/experimental/spannerio.py", line 1251, in process
> E             self.service_metric.call('ok')
> E         AttributeError: 'NoneType' object has no attribute 'call'
> E         
> E         During handling of the above exception, another exception occurred:
> E         
> E         Traceback (most recent call last):
> E           File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 646, in do_work
> E             work_executor.execute()
> E           File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 210, in execute
> E             op.finish()
> E           File "apache_beam/runners/worker/operations.py", line 736, in apache_beam.runners.worker.operations.DoOperation.finish
> E           File "apache_beam/runners/worker/operations.py", line 738, in apache_beam.runners.worker.operations.DoOperation.finish
> E           File "apache_beam/runners/worker/operations.py", line 739, in apache_beam.runners.worker.operations.DoOperation.finish
> E           File "apache_beam/runners/common.py", line 1253, in apache_beam.runners.common.DoFnRunner.finish
> E           File "apache_beam/runners/common.py", line 1234, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
> E           File "apache_beam/runners/common.py", line 1265, in apache_beam.runners.common.DoFnRunner._reraise_augmented
> E           File "apache_beam/runners/common.py", line 1232, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
> E           File "apache_beam/runners/common.py", line 475, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
> E           File "apache_beam/runners/common.py", line 480, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
> E           File "apache_beam/runners/common.py", line 1401, in apache_beam.runners.common._OutputProcessor.finish_bundle_outputs
> E           File "apache_beam/runners/worker/operations.py", line 215, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
> E           File "apache_beam/runners/worker/operations.py", line 707, in apache_beam.runners.worker.operations.DoOperation.process
> E           File "apache_beam/runners/worker/operations.py", line 708, in apache_beam.runners.worker.operations.DoOperation.process
> E           File "apache_beam/runners/common.py", line 1200, in apache_beam.runners.common.DoFnRunner.process
> E           File "apache_beam/runners/common.py", line 1281, in apache_beam.runners.common.DoFnRunner._reraise_augmented
> E           File "apache_beam/runners/common.py", line 1198, in apache_beam.runners.common.DoFnRunner.process
> E           File "apache_beam/runners/common.py", line 537, in apache_beam.runners.common.SimpleInvoker.invoke_process
> E           File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/experimental/spannerio.py", line 1251, in process
> E             self.service_metric.call('ok')
> E         AttributeError: 'NoneType' object has no attribute 'call' [while running 'WriteToSpanner/Writing to spanner']
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)