You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "AmirF194 (via GitHub)" <gi...@apache.org> on 2026/08/31 22:07:28 UTC
[PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
AmirF194 opened a new pull request, #39943:
URL: https://github.com/apache/beam/pull/39943
`PickleCoder.as_deterministic_coder()` and `_MemoizingPickleCoder.as_deterministic_coder()` both construct `FastPrimitivesCoder` with a `requires_deterministic` keyword argument:
```python
def as_deterministic_coder(self, step_label, error_message=None):
return FastPrimitivesCoder(self, requires_deterministic=step_label)
```
`FastPrimitivesCoder.__init__(self, fallback_coder=PickleCoder())` has never accepted that argument, so both raise `TypeError` unconditionally. The bug goes back to commit 9be70c9861, which changed `PickleCoder.as_deterministic_coder` from the working `DeterministicFastPrimitivesCoder(self, step_label)` to this broken call, in the same diff that added a `requires_deterministic` kwarg to a different class (`coder_impl.FastPrimitivesCoderImpl`, the Cython impl, not the `coders.FastPrimitivesCoder` wrapper referenced here). `_MemoizingPickleCoder` later copied the already-broken line.
The invariant `as_deterministic_coder()` is supposed to hold, that it always returns a working deterministic coder rather than raising, is exactly what `FastPrimitivesCoder`'s own `as_deterministic_coder` already does by delegating to `_update_compatible_deterministic_fast_primitives_coder`. This PR routes both call sites through that same helper instead.
A reachable consumer: `GroupByEncryptedKey.expand()` calls `coder.as_deterministic_coder(...)` inside `except ValueError`, expecting a non-deterministic key to log a warning. When the coder resolves to `PickleCoder`/`_MemoizingPickleCoder`, the uncaught `TypeError` crashes pipeline construction instead.
Verified:
- `PickleCoder().as_deterministic_coder('x')` and `_MemoizingPickleCoder().as_deterministic_coder('x')` raise `TypeError` on `master`, confirmed live against the installed `apache-beam` package (byte-identical `coders.py`) and against this branch.
- New `PickleCoderTest.test_as_deterministic_coder` fails on `master`, passes on this branch, both runs in the same container.
- `coders_test.py`, `typecoders_test.py`, and `fast_coders_test.py` (the standard-coder correctness suite, all classes including `FastPrimitivesCoder`/`DeterministicFastPrimitivesCoder`) pass unchanged.
- `ruff check` and `yapf --diff` are clean on both changed files.
- Not checked: the Cython (`coder_impl`) fast paths, since this fix is confined to the plain-Python `coders.py` wrapper layer and does not touch `coder_impl`.
Fixes #39942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm merged PR #39943:
URL: https://github.com/apache/beam/pull/39943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39943:
URL: https://github.com/apache/beam/pull/39943#issuecomment-5579954509
Ok - I'll remind @AmirF194 after tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "AmirF194 (via GitHub)" <gi...@apache.org>.
AmirF194 commented on PR #39943:
URL: https://github.com/apache/beam/pull/39943#issuecomment-5579943189
remind me after tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #39943:
URL: https://github.com/apache/beam/pull/39943#discussion_r4017241636
##########
sdks/python/apache_beam/coders/coders.py:
##########
@@ -914,7 +915,8 @@ def _create_impl(self):
lambda x: dumps(x, protocol), pickle.loads)
def as_deterministic_coder(self, step_label, error_message=None):
- return FastPrimitivesCoder(self, requires_deterministic=step_label)
+ return _update_compatible_deterministic_fast_primitives_coder(
Review Comment:
Returning DeterministicFastPrimitivesCoder will fail at runtime for arbitrary classes.
https://github.com/apache/beam/blob/42663f32154b454c8bc64dd312373b3138c84f1c/sdks/python/apache_beam/coders/coder_impl.py#L486
Maybe add a warning like
```
PickleCoder was registered for a key type in '%s', but the runner requires a
deterministic key encoding and pickle is not deterministic. Keys in this step
will be encoded with the deterministic fallback coder instead of pickle. That
coder supports primitives, containers, protobuf messages, frozen dataclasses,
NamedTuples, enums, and classes defining both __getstate__ and __setstate__.
Any other key type will fail at encode time. If your key type is not one of
these, register a deterministic custom Coder for it or add a type hint so the
default coder is used.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39943:
URL: https://github.com/apache/beam/pull/39943#issuecomment-5776554691
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damccorm for label python.
This pull request likely touches a core component ("core" label). Please review with scrutiny.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39943:
URL: https://github.com/apache/beam/pull/39943#issuecomment-5516780051
Assigning reviewers:
R: @claudevdm for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "AmirF194 (via GitHub)" <gi...@apache.org>.
AmirF194 commented on PR #39943:
URL: https://github.com/apache/beam/pull/39943#issuecomment-5507010069
Run Python_Integration PreCommit 3.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "AmirF194 (via GitHub)" <gi...@apache.org>.
AmirF194 commented on PR #39943:
URL: https://github.com/apache/beam/pull/39943#issuecomment-5685263522
Good point, pushed fa9623c. Added a warning at both call sites (PickleCoder and _MemoizingPickleCoder) using close to your wording, logged before the fallback coder is constructed. Ran the full coders_test.py suite plus a manual check that the warning fires with the right step label; 16 passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39943:
URL: https://github.com/apache/beam/pull/39943#issuecomment-5680195274
Reminder, please take a look at this pr: @claudevdm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix PickleCoder.as_deterministic_coder() raising TypeError [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39943:
URL: https://github.com/apache/beam/pull/39943#issuecomment-5486040788
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org