You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "apraga (via GitHub)" <gi...@apache.org> on 2025/02/04 21:09:39 UTC
[I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
apraga opened a new issue, #33854:
URL: https://github.com/apache/beam/issues/33854
### What happened?
When packaging apache beam 2.62 in nix ( https://github.com/NixOS/nixpkgs/pull/379201 ), around 70 tests fail with a single error ` AttributeError: 'MaybeReshuffle' object has no attribute 'side_inputs'`.
The list is below. Thanks for your help
```
apache_beam/examples/snippets/transforms/aggregation/approximatequantiles_test.py
apache_beam/examples/snippets/transforms/aggregation/approximateunique_test.py
apache_beam/examples/snippets/transforms/aggregation/batchelements_test.py
apache_beam/examples/snippets/transforms/aggregation/cogroupbykey_test.py
apache_beam/examples/snippets/transforms/aggregation/combineglobally_test.py
apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py
apache_beam/examples/snippets/transforms/aggregation/combinevalues_test.py
apache_beam/examples/snippets/transforms/aggregation/count_test.py
apache_beam/examples/snippets/transforms/aggregation/distinct_test.py
apache_beam/examples/snippets/transforms/aggregation/groupbykey_test.py
apache_beam/examples/snippets/transforms/aggregation/groupintobatches_test.py
apache_beam/examples/snippets/transforms/aggregation/latest_test.py
apache_beam/examples/snippets/transforms/aggregation/max_test.py
apache_beam/examples/snippets/transforms/aggregation/mean_test.py
apache_beam/examples/snippets/transforms/aggregation/min_test.py
apache_beam/examples/snippets/transforms/aggregation/sample_test.py
apache_beam/examples/snippets/transforms/aggregation/sum_test.py
apache_beam/examples/snippets/transforms/aggregation/tolist_test.py
apache_beam/examples/snippets/transforms/aggregation/top_test.py
apache_beam/examples/snippets/transforms/elementwise/filter_test.py
apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
apache_beam/examples/snippets/transforms/elementwise/keys_test.py
apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
apache_beam/examples/snippets/transforms/elementwise/map_test.py
apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
apache_beam/examples/snippets/transforms/elementwise/partition_test.py
apache_beam/examples/snippets/transforms/elementwise/regex_test.py
apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
apache_beam/examples/snippets/transforms/elementwise/values_test.py
apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
apache_beam/examples/snippets/transforms/other/create_test.py
apache_beam/examples/snippets/transforms/other/flatten_test.py
apache_beam/examples/snippets/transforms/other/window_test.py
apache_beam/examples/snippets/util_test.py
apache_beam/io/avroio_test.py
apache_beam/io/concat_source_test.py
apache_beam/io/filebasedsink_test.py
apache_beam/io/filebasedsource_test.py
apache_beam/io/fileio_test.py
apache_beam/io/mongodbio_test.py
apache_beam/io/parquetio_test.py
apache_beam/io/sources_test.py
apache_beam/io/textio_test.py
apache_beam/io/tfrecordio_test.py
apache_beam/metrics/metric_test.py
apache_beam/ml/inference/base_test.py
apache_beam/ml/inference/sklearn_inference_test.py
apache_beam/ml/inference/utils_test.py
apache_beam/ml/rag/chunking/base_test.py
apache_beam/pipeline_test.py
apache_beam/runners/direct/direct_runner_test.py
apache_beam/runners/direct/sdf_direct_runner_test.py
apache_beam/runners/interactive/interactive_beam_test.py
apache_beam/runners/interactive/interactive_runner_test.py
apache_beam/runners/interactive/non_interactive_runner_test.py
apache_beam/runners/interactive/recording_manager_test.py
apache_beam/runners/portability/fn_api_runner/translations_test.py
apache_beam/runners/portability/fn_api_runner/trigger_manager_test.py
apache_beam/runners/portability/stager_test.py
apache_beam/testing/synthetic_pipeline_test.py
apache_beam/testing/test_stream_test.py
apache_beam/testing/util_test.py
apache_beam/transforms/combiners_test.py
apache_beam/transforms/core_test.py
apache_beam/transforms/create_test.py
apache_beam/transforms/deduplicate_test.py
apache_beam/transforms/periodicsequence_test.py
apache_beam/transforms/ptransform_test.py
apache_beam/transforms/sideinputs_test.py
apache_beam/transforms/stats_test.py
apache_beam/transforms/transforms_keyword_only_args_test.py
apache_beam/transforms/trigger_test.py
apache_beam/transforms/userstate_test.py
apache_beam/transforms/util_test.py
apache_beam/transforms/write_ptransform_test.py
```
### Issue Failure
Failure: Test is flaky
### Issue Priority
Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad closed issue #33854: [Failing Test]: AttributeError for MaybeReshuffle in Python SDK
URL: https://github.com/apache/beam/issues/33854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "GaetanLepage (via GitHub)" <gi...@apache.org>.
GaetanLepage commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-2817847875
Hello,
This issue was opened in the context of `apache-beam` packaging in the [nixpkgs](https://github.com/NixOS/nixpkgs/) software repository.
We (package maintainers) are observing that these tests fail and we have thus disabled them. We run the test suite of all python packages to ensure that we have packaged them correctly.
> Can you please try clearing your dependency cache and reinstalling beam + dependencies ?
Due to the nature of the nix builder itself, there is no polluting dependency cache to clear. The environment in which the package is tested only contains the dependencies that we have explicitly listed.
> If it does not work, try with some different python version. (For example 3.9.2)
We currently support the python 3.12 and python 3.13 interpreters and are ensuring that all python packages function correctly against these versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "smblsm (via GitHub)" <gi...@apache.org>.
smblsm commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-2813552864
Wondering if there's any more info on this, because I'm running into it to when upgrading from 2.60.0 to 2.64.0. Any test using TestPipeline fails with `AttributeError: 'MaybeReshuffle' object has no attribute 'side_inputs'`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-3325074980
https://github.com/apache/beam/pull/36251 to fix that. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "TanuSharma2511 (via GitHub)" <gi...@apache.org>.
TanuSharma2511 commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-2743621837
One potential reason for this error is dependencies conflict or not proper installation of any dependency. You should see more information in stack trace. In my case, pyarrow was creating issue. So, I re-install it: pip install pyarrow and error got resolved.
Please check the stack trace. You will get the idea which dependency is creating issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-3312193332
@GaetanLepage @apraga starting from 2.65.0, Beam switches to cloud pickle. Can you try the latest Beam?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-3321984118
This looks much easier to fix. I got https://github.com/apache/beam/pull/36238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad closed issue #33854: [Failing Test]: AttributeError for MaybeReshuffle in Python SDK
URL: https://github.com/apache/beam/issues/33854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "apraga (via GitHub)" <gi...@apache.org>.
apraga commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-2814008753
Hi,
Error log is attached if that can help.
[error.log](https://github.com/user-attachments/files/19800965/error.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "GaetanLepage (via GitHub)" <gi...@apache.org>.
GaetanLepage commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-3322975031
Thanks! It seems to have fixed the failing the first occurrence. However, it fails further down:
```
# We may need to merge the hints with environment-provided hints here
# once environment is a first-class citizen in Beam graph and we have
# access to actual environment, not just an id.
self.resource_hints = dict(
> transform.get_resource_hints()) if transform else {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
} # type: Dict[str, bytes]
E AttributeError: 'MaybeReshuffle' object has no attribute 'get_resource_hints'
apache_beam/pipeline.py:1239: AttributeError
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-3292390906
I happened to see this issue (and a few other related ones mentioned in https://github.com/NixOS/nixpkgs/pull/379201) while scanning through our backlog. Thanks for bring this up.
We would like to help but we are not familiar with the packaging system/process in nix. If you can provide steps to reproduce (e.g. maybe running your tests in a docker environment), then I can help to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "TanuSharma2511 (via GitHub)" <gi...@apache.org>.
TanuSharma2511 commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-2817678240
Can you please try clearing your dependency cache and reinstalling beam + dependencies ? If it does not work, try with some different python version. (For example 3.9.2)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-3367434918
Please re-open this if this still causes the issues for NixOS/nixpkgs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: AttributeError for MaybeReshuffle in Python SDK [beam]
Posted by "GaetanLepage (via GitHub)" <gi...@apache.org>.
GaetanLepage commented on issue #33854:
URL: https://github.com/apache/beam/issues/33854#issuecomment-3321807582
On apache-beam 2.68.0, we have one occurence left:
```
______________ PipelineTest.test_runner_overrides_default_pickler ______________
[gw40] linux -- Python 3.13.7 /nix/store/829wb290i87wngxlh404klwxql5v18p4-python3-3.13.7/bin/python3.13
self = <apache_beam.pipeline_test.PipelineTest testMethod=test_runner_overrides_default_pickler>
mock_info = <MagicMock name='info' id='140719519900944'>
@mock.patch('logging.info')
def test_runner_overrides_default_pickler(self, mock_info):
with mock.patch.object(PipelineRunner,
'default_pickle_library_override') as mock_fn:
mock_fn.return_value = 'dill'
> with TestPipeline() as pipeline:
^^^^^^^^^^^^^^
apache_beam/pipeline_test.py:184:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/pipeline.py:666: in __exit__
self.result = self.run()
^^^^^^^^^^
apache_beam/testing/test_pipeline.py:118: in run
result = super().run(
apache_beam/pipeline.py:613: in run
return Pipeline.from_runner_api(
apache_beam/pipeline.py:1103: in from_runner_api
p.transforms_stack = [context.transforms.get_by_id(root_transform_id)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
apache_beam/runners/pipeline_context.py:106: in get_by_id
self._id_to_obj[id] = self._obj_type.from_runner_api(
apache_beam/pipeline.py:1530: in from_runner_api
part = context.transforms.get_by_id(transform_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
apache_beam/runners/pipeline_context.py:106: in get_by_id
self._id_to_obj[id] = self._obj_type.from_runner_api(
apache_beam/pipeline.py:1530: in from_runner_api
part = context.transforms.get_by_id(transform_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
apache_beam/runners/pipeline_context.py:106: in get_by_id
self._id_to_obj[id] = self._obj_type.from_runner_api(
apache_beam/pipeline.py:1530: in from_runner_api
part = context.transforms.get_by_id(transform_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
apache_beam/runners/pipeline_context.py:106: in get_by_id
self._id_to_obj[id] = self._obj_type.from_runner_api(
apache_beam/pipeline.py:1516: in from_runner_api
result = AppliedPTransform(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = AppliedPTransform(assert_that/Create/MaybeReshuffle, MaybeReshuffle)
parent = None
transform = <apache_beam.transforms.core.Create.expand.<locals>.MaybeReshuffle object at 0x7ffbd03a6900>
full_label = 'assert_that/Create/MaybeReshuffle'
main_inputs = {'None': <PCollection[assert_that/Create/FlatMap(<lambda at core.py:4094>).None] at 0x7ffbd041bdd0>}
environment_id = None
annotations = {'python_type': b'apache_beam.transforms.core.Create.expand.<locals>.MaybeReshuffle'}
def __init__(
self,
parent, # type: Optional[AppliedPTransform]
transform, # type: Optional[ptransform.PTransform]
full_label, # type: str
main_inputs, # type: Optional[Mapping[str, Union[pvalue.PBegin, pvalue.PCollection]]]
environment_id, # type: Optional[str]
annotations, # type: Optional[Dict[str, bytes]]
):
# type: (...) -> None
self.parent = parent
self.transform = transform
# Note that we want the PipelineVisitor classes to use the full_label,
# inputs, side_inputs, and outputs fields from this instance instead of the
# ones of the PTransform instance associated with it. Doing this permits
# reusing PTransform instances in different contexts (apply() calls) without
# any interference. This is particularly useful for composite transforms.
self.full_label = full_label
self.main_inputs = dict(main_inputs or {})
> self.side_inputs = tuple() if transform is None else transform.side_inputs
^^^^^^^^^^^^^^^^^^^^^
E AttributeError: 'MaybeReshuffle' object has no attribute 'side_inputs'
apache_beam/pipeline.py:1229: AttributeError
```
https://github.com/NixOS/nixpkgs/pull/445349
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org