You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/06/01 17:24:00 UTC
[jira] [Work logged] (BEAM-14481) ScopedState can not be re-entered (regression caused by Batched DoFn worker changes)
[ https://issues.apache.org/jira/browse/BEAM-14481?focusedWorklogId=777034&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777034 ]
ASF GitHub Bot logged work on BEAM-14481:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jun/22 17:23
Start Date: 01/Jun/22 17:23
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on PR #17761:
URL: https://github.com/apache/beam/pull/17761#issuecomment-1143910391
It still failed, it looks like we [cache ScopedState instances](https://github.com/apache/beam/blob/ca33943808c56ce634c92eb85f865285c71ee048/sdks/python/apache_beam/runners/worker/statesampler.py#L169-L170) and there are cases where they are re-used across operations. PreCommit is failing in the Dataflow worker now:
```
> self)
E apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
E Traceback (most recent call last):
E File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 646, in do_work
E work_executor.execute()
E File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 210, in execute
E op.finish()
E File "apache_beam/runners/worker/operations.py", line 1269, in apache_beam.runners.worker.operations.PGBKCVOperation.finish
E File "apache_beam/runners/worker/operations.py", line 1272, in apache_beam.runners.worker.operations.PGBKCVOperation.finish
E File "apache_beam/runners/worker/operations.py", line 1295, in apache_beam.runners.worker.operations.PGBKCVOperation.output_key
E File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
E File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
E File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
E File "dataflow_worker/shuffle_operations.py", line 214, in dataflow_worker.shuffle_operations.ReifyTimestampAndWindowsOperation.process
E File "dataflow_worker/shuffle_operations.py", line 218, in dataflow_worker.shuffle_operations.ReifyTimestampAndWindowsOperation.process
E File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
E File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
E File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
E File "dataflow_worker/shuffle_operations.py", line 177, in dataflow_worker.shuffle_operations.ShuffleWriteOperation.process
E File "apache_beam/runners/worker/statesampler_fast.pyx", line 235, in apache_beam.runners.worker.statesampler_fast.ScopedState.__enter__
E File "apache_beam/runners/worker/statesampler_fast.pyx", line 236, in apache_beam.runners.worker.statesampler_fast.ScopedState.__enter__
E AssertionError: Attempted to re-enter ScopedState[s05-s4-process-msecs, 235098419], which is unnecessary and unsupported
```
Maybe we need to make ScopedState support re-entering...
Issue Time Tracking
-------------------
Worklog Id: (was: 777034)
Time Spent: 4h (was: 3h 50m)
> ScopedState can not be re-entered (regression caused by Batched DoFn worker changes)
> ------------------------------------------------------------------------------------
>
> Key: BEAM-14481
> URL: https://issues.apache.org/jira/browse/BEAM-14481
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: P2
> Time Spent: 4h
> Remaining Estimate: 0h
>
> We've observed some regressions in internal pipelines. We've narrowed the root cause down to [https://github.com/apache/beam/pull/17384] - the worker changes for Batched DoFns.
> I still don't understand the root cause, but given this change can lead to a regression in pipelines that don't use the feature I think we should roll it back.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)