You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "raams95 (via GitHub)" <gi...@apache.org> on 2026/10/20 05:52:41 UTC
[PR] [Python] Infer the right type for slices of fixed-size tuples [beam]
raams95 opened a new pull request, #40680:
URL: https://github.com/apache/beam/pull/40680
Fixes #20899
Type inference gives a slice of a fixed-size tuple the type of the whole tuple. For an element of type `Tuple[str, str, int]`, `row[1:]` is inferred as `Tuple[str, str, int]` instead of `Tuple[str, int]`. This breaks pipelines, not only the reported `element_type`. Dropping the first field before combining fails at construction:
```python
rows = [('day1', 'a', 1), ('day2', 'a', 2), ('day1', 'b', 3)]
with beam.Pipeline() as p:
_ = (
p
| beam.Create(rows).with_output_types(Tuple[str, str, int])
| 'DropDate' >> beam.Map(lambda row: row[1:])
| beam.CombinePerKey(sum))
```
```
TypeCheckError: The transform 'CombinePerKey(sum)' requires PCollections of type 'Tuple[TypeVariable[K], Any]' but was applied to a PCollection of type 'Tuple[<class 'str'>, <class 'str'>, <class 'int'>]' (produced by the transform 'DropDate').
```
When the sliced PCollection is encoded, for example before a `Reshuffle`, the coder for the whole tuple is used and the pipeline fails with `TypeError: '<' not supported between instances of 'str' and 'int'` for `row[1:]`, or `IndexError: tuple index out of range` for `row[:2]`.
With this change, a slice of a fixed-size `Tuple` with constant bounds is inferred as the tuple of the selected element types (`Tuple[str, int]` above, `Tuple[float, str, int]` for `[::-1]` on `Tuple[int, str, float]`). A slice with bounds that aren't constants, such as `row[i:]`, is inferred as `Tuple[Union[...], ...]` of the element types, in line with a variable index (`row[i]` is inferred as `Union[...]`) and tuple unpacking (`(*a, *b)`). Slices of lists, `Tuple[T, ...]` and `str` keep the type of the sliced value, as before.
Slices compile to different bytecode depending on the Python version: `BUILD_SLICE` followed by a subscript on 3.10 and 3.11 and for slices with a step, `BINARY_SLICE` on 3.12 and later, and a constant `slice` object on 3.14. `build_slice` now keeps constant bounds as a `Const` slice, and `binary_subscr` and `binary_slice` share one helper, so all versions infer the same types.
Tests:
- `testGetItemSlice` in `trivial_inference_test.py` has new cases for fixed-size tuples.
- `test_inferred_type_of_tuple_slice` in `typed_pipeline_test.py` runs the pipeline above.
Both fail before the change and pass after it on Python 3.10, 3.11, 3.12, 3.13 and 3.14.
Testing:
- `pytest apache_beam/typehints/trivial_inference_test.py apache_beam/typehints/typed_pipeline_test.py` on Python 3.10, 3.11, 3.12, 3.13 and 3.14
- `pytest apache_beam/typehints apache_beam/transforms apache_beam/coders apache_beam/pipeline_test.py apache_beam/dataframe/transforms_test.py apache_beam/yaml/yaml_mapping_test.py apache_beam/yaml/yaml_udf_test.py` on Python 3.12
- `yapf --diff` and `ruff check` on the changed files
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example: `addresses #123` or `fixes #456`).
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). (Not needed, small change.)
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Infer the right type for slices of fixed-size tuples [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40680:
URL: https://github.com/apache/beam/pull/40680#issuecomment-6285227595
Assigning reviewers:
R: @jrmccluskey for label python.
Backup reviewer: damccorm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)
This pull request likely touches a core component ("core" label). Please review with scrutiny.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Infer the right type for slices of fixed-size tuples [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #40680:
URL: https://github.com/apache/beam/pull/40680#discussion_r4326551792
##########
CHANGES.md:
##########
@@ -104,6 +104,7 @@
* (Python) DaskRunner now preserves outputs emitted by `DoFn.finish_bundle()` instead of silently dropping them ([#40519](https://github.com/apache/beam/issues/40519)).
* (Python) BigQuery streaming inserts now reset the retry budget and backoff for each batch instead of sharing them across a bundle ([#39995](https://github.com/apache/beam/pull/39995)).
* (Python) State write failures reported by portable runners now fail the bundle instead of silently losing state updates ([#39992](https://github.com/apache/beam/pull/39992)).
+* (Python) Type inference now gives the right type for a slice of a fixed-size tuple, such as `row[1:]` in `beam.Map(lambda row: row[1:])`. The slice used to keep the type of the whole tuple, which could make a following `CombinePerKey` fail with a `TypeCheckError` or encode elements with the wrong coder ([#20899](https://github.com/apache/beam/issues/20899)).
Review Comment:
This is a hair verbose, let's streamline a bit and let the context live in the linked bug.
```suggestion
* (Python) Type inference now gives the right type for a slice of a fixed-size tuple, such as `row[1:]` in `beam.Map(lambda row: row[1:])`. ([#20899](https://github.com/apache/beam/issues/20899)).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Infer the right type for slices of fixed-size tuples [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #40680:
URL: https://github.com/apache/beam/pull/40680#issuecomment-6284821400
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/40680?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `63.15789%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.02%. Comparing base ([`76f6e6d`](https://app.codecov.io/gh/apache/beam/commit/76f6e6d7c0c1b2ce582c73d068ad9260cf15c5f3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d545485`](https://app.codecov.io/gh/apache/beam/commit/d5454852e20558c33fc9ee8a6a87c448ea8018e8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 242 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/40680?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [sdks/python/apache\_beam/typehints/opcodes.py](https://app.codecov.io/gh/apache/beam/pull/40680?src=pr&el=tree&filepath=sdks%2Fpython%2Fapache_beam%2Ftypehints%2Fopcodes.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | 63.15% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/40680?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #40680 +/- ##
=============================================
+ Coverage 56.16% 80.02% +23.85%
=============================================
Files 1124 490 -634
Lines 178336 82825 -95511
Branches 1489 0 -1489
=============================================
- Hits 100170 66278 -33892
+ Misses 75645 16547 -59098
+ Partials 2521 0 -2521
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/40680/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [go](https://app.codecov.io/gh/apache/beam/pull/40680/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java](https://app.codecov.io/gh/apache/beam/pull/40680/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [python](https://app.codecov.io/gh/apache/beam/pull/40680/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.02% <63.15%> (+0.46%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/40680?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org