You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "s3847243 (via GitHub)" <gi...@apache.org> on 2025/04/20 04:52:01 UTC
[PR] Add withReshuffle(boolean) option to FileIO.matchAll() [beam]
s3847243 opened a new pull request, #34677:
URL: https://github.com/apache/beam/pull/34677
### What
This PR adds a new method `withReshuffle(boolean)` to `FileIO.matchAll()` to allow disabling the automatic reshuffling step (`Reshuffle.viaRandomKey()`).
### Why
Currently, `FileIO.matchAll()` always applies a `Reshuffle` step. While this improves performance for wildcard patterns that expand into many files, it is not ideal when processing a large static list of file paths (e.g., 1M+). In such cases, reshuffling can block downstream fusion and autoscaling.
This feature allows advanced users to opt out of reshuffling to improve performance and fusion behavior.
### How
- Added a `getReshuffle()` property to the `MatchAll` AutoValue class, defaulting to `true`.
- Updated the `expand()` method to conditionally apply the reshuffle based on the property.
- Added `withReshuffle(boolean)` for API access.
- Updated `FileIOTest` to verify the reshuffle toggle behavior.
### Fixes
Fixes: #33330
---
- [x] This addresses an open issue (`fixes #33330`)
- [x] Tested and verified the updated behavior
- [ ] I will update `CHANGES.md` upon approval if necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add withReshuffle(boolean) option to FileIO.matchAll() [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34677:
URL: https://github.com/apache/beam/pull/34677#issuecomment-2987988540
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add withReshuffle(boolean) option to FileIO.matchAll() [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34677:
URL: https://github.com/apache/beam/pull/34677#issuecomment-2817034141
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org