You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tvalentyn (via GitHub)" <gi...@apache.org> on 2023/11/20 18:12:08 UTC
[I] [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages [beam]
tvalentyn opened a new issue, #27330:
URL: https://github.com/apache/beam/issues/27330
### What happened?
We suspect that an upgrade to `protobuf==4.x.x` in Beam SDK & worker containers (https://github.com/apache/beam/pull/24599) introduced a failure mode in Python streaming pipelines, where Python process now sometimes crash with `AttributeError` or segmentation faults.
Example errors:
```
File "/usr/local/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1655, in _create_pardo_operation
output_tags = list(transform_proto.outputs.keys())
AttributeError: 'tuple' object has no attribute 'keys'
```
```
File "/usr/local/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1314, in get_output_coders
pcoll_id in transform_proto.outputs.items()
AttributeError: 'function' object has no attribute 'items'
```
```
File "/usr/local/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 819, in wrapper
result = cache[args] = func(*args)
File "/usr/local/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 979, in topological_height
for pcoll in descriptor.transforms[transform_id].outputs.values()
AttributeError: 'function' object has no attribute 'values'
```
```
File "/usr/local/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 819, in wrapper
result = cache[args] = func(*args)
Default Python SDK image for environment is apache/beam_python3.10_sdk:2.47.0.dev
File "/usr/local/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 979, in topological_height
for pcoll in descriptor.transforms[transform_id].outputs.values()
AttributeError: 'traceback' object has no attribute 'values'
```
The pipelines usually recover after the process crash but may cause delays.
Exact nature for how this happens is unclear at the moment. It may be a preexisting latent error, that now received a different manifestation.
While this is investigated, as a precaution, it would be advisable to use 2.46.0 or lower Beam SDKs.
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #27330: [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #27330:
URL: https://github.com/apache/beam/issues/27330#issuecomment-1726312911
we expect it to be resolved in 2.51.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #27330: [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #27330:
URL: https://github.com/apache/beam/issues/27330#issuecomment-1710489599
@victorrgez Thank you for the feedback. The attribute error issues should we fixed once we upgrade to the upcoming release of protobuf library.
Python 3.10-specific crashes should be fixed with https://github.com/apache/beam/pull/28355.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] victorrgez commented on issue #27330: [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages
Posted by "victorrgez (via GitHub)" <gi...@apache.org>.
victorrgez commented on issue #27330:
URL: https://github.com/apache/beam/issues/27330#issuecomment-1653402568
We have experienced the same issue in loop for around 5 minutes in Python 3.11 in Dataflow and then it got solved automatically:
```
"Error processing instruction process_bundle-12173-17596. Original traceback is
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 295, in _execute
response = task()
^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 370, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 629, in do_instruction
return getattr(self, request_type)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 660, in process_bundle
bundle_processor = self.bundle_processor_cache.get(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py", line 491, in get
processor = bundle_processor.BundleProcessor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 903, in __init__
self.ops = self.create_execution_tree(self.process_bundle_descriptor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 985, in create_execution_tree
get_operation(transform_id))) for transform_id in sorted(
^^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 818, in wrapper
result = cache[args] = func(*args)
^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 976, in topological_height
return 1 + max([0] + [
^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 977, in <listcomp>
topological_height(consumer)
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 818, in wrapper
result = cache[args] = func(*args)
^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py", line 978, in topological_height
for pcoll in descriptor.transforms[transform_id].outputs.values()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'values'
"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python SDK sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #27330: [Bug]: Python SDK sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages
URL: https://github.com/apache/beam/issues/27330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #27330:
URL: https://github.com/apache/beam/issues/27330#issuecomment-1819597583
There is some evidence that this issue is not fully resolved, so I am reopening it and will look closer to see if I can repro it reliably for further investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #27330: [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #27330:
URL: https://github.com/apache/beam/issues/27330#issuecomment-1631397239
We suspect the issue affects only Python 3.10 SDK. Immediate workaround may be to use Python 3.9 or Python 3.11. We are working on confirming the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python SDK sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #27330:
URL: https://github.com/apache/beam/issues/27330#issuecomment-1862569177
We expect this issue to be resolved in 2.53.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #27330: [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #27330:
URL: https://github.com/apache/beam/issues/27330#issuecomment-1725635590
Any update on this P1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn closed issue #27330: [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #27330: [Bug]: Python SDK crashes sometimes crashes in streaming jobs running on 2.47.0+ SDK with AttributeError messages
URL: https://github.com/apache/beam/issues/27330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org