You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "lostluck (via GitHub)" <gi...@apache.org> on 2023/04/29 00:37:30 UTC
[GitHub] [beam] lostluck opened a new pull request, #26484: [prism] Add dynamic channel splits.
lostluck opened a new pull request, #26484:
URL: https://github.com/apache/beam/pull/26484
This PR adds basic progress tracking and dynamic splitting to prism.
This is basic support to pass basic separation harness tests with additional configuration to come at a later time.
The split is always to the remaining third of elements in an executing bundle, and only if progress doesn't advance for a bundle between progress requests.
TODOs:
* Add a "close sentinel" check to ensure that consecutive elements become unblocked.
* Add an SDF check to ensure that sub elements are split as well.
* See if we can do fewer active Goroutines
* Spawn bundle processing goroutines on demand only.
* Don't spawn a separate progress & split tracking goroutine, which means changing the data blocking logic for *B.ProcessOn
* Add "provision" server to handle capabilities and worker startup.
* Add monitoring short id tests.
Adds support for monitoring_short_id's runner capability.
Fixes memory leak in worker bundle tracking, where the bundle and data were not getting GCd.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #26484: [prism] Add dynamic channel & sub element splits.
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #26484:
URL: https://github.com/apache/beam/pull/26484#issuecomment-1531953342
Thanks for the review!
Yeah, this definitely validates they work. The lack of them being used Dataflow side is a service side restriction that I need to validate against older versions to get it lifted. Might be "clear" as fare as 2.33 if we're lucky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #26484: [prism] Add dynamic channel & sub element splits.
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck merged PR #26484:
URL: https://github.com/apache/beam/pull/26484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #26484: [prism] Add dynamic channel splits.
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26484:
URL: https://github.com/apache/beam/pull/26484#issuecomment-1530050554
## [Codecov](https://codecov.io/gh/apache/beam/pull/26484?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#26484](https://codecov.io/gh/apache/beam/pull/26484?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e3e56b) into [master](https://codecov.io/gh/apache/beam/commit/955a04eac5ff4e7d3c3c166df521d672fe5f59c0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (955a04e) will **increase** coverage by `9.17%`.
> The diff coverage is `91.33%`.
> :exclamation: Current head 5e3e56b differs from pull request most recent head 874ca59. Consider uploading reports for the commit 874ca59 to get more accurate results
```diff
@@ Coverage Diff @@
## master #26484 +/- ##
==========================================
+ Coverage 71.95% 81.13% +9.17%
==========================================
Files 752 469 -283
Lines 101709 67420 -34289
==========================================
- Hits 73185 54700 -18485
+ Misses 27028 12720 -14308
+ Partials 1496 0 -1496
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `?` | |
| python | `81.13% <91.33%> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/26484?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/jdbc.py](https://codecov.io/gh/apache/beam/pull/26484?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vamRiYy5weQ==) | `79.16% <ø> (ø)` | |
| [sdks/python/setup.py](https://codecov.io/gh/apache/beam/pull/26484?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vc2V0dXAucHk=) | `0.00% <ø> (ø)` | |
| [sdks/python/apache\_beam/yaml/yaml\_transform.py](https://codecov.io/gh/apache/beam/pull/26484?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0veWFtbC95YW1sX3RyYW5zZm9ybS5weQ==) | `86.25% <90.67%> (ø)` | |
| [sdks/python/apache\_beam/yaml/yaml\_provider.py](https://codecov.io/gh/apache/beam/pull/26484?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0veWFtbC95YW1sX3Byb3ZpZGVyLnB5) | `66.39% <93.10%> (ø)` | |
| [sdks/python/apache\_beam/transforms/display.py](https://codecov.io/gh/apache/beam/pull/26484?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9kaXNwbGF5LnB5) | `94.03% <100.00%> (ø)` | |
... and [295 files with indirect coverage changes](https://codecov.io/gh/apache/beam/pull/26484/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26484: [prism] Add dynamic channel splits.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26484:
URL: https://github.com/apache/beam/pull/26484#issuecomment-1530652864
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @jrmccluskey for label go.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #26484: [prism] Add dynamic channel splits.
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #26484:
URL: https://github.com/apache/beam/pull/26484#issuecomment-1530586545
Something went odd when I rebased (it includes all the master commits), but the diff is correct with the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26484: [prism] Add dynamic channel splits.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26484:
URL: https://github.com/apache/beam/pull/26484#issuecomment-1531629726
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @riteshghorse for label go.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on pull request #26484: [prism] Add dynamic channel splits.
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26484:
URL: https://github.com/apache/beam/pull/26484#issuecomment-1531627386
assign to next reviewer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a diff in pull request #26484: [prism] Add dynamic channel & sub element splits.
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #26484:
URL: https://github.com/apache/beam/pull/26484#discussion_r1182826993
##########
sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go:
##########
@@ -40,47 +40,66 @@ type B struct {
// MultiMapSideInputData is a map from transformID, to inputID, to window, to data key, to data values.
MultiMapSideInputData map[string]map[string]map[typex.Window]map[string][][]byte
- // OutputCount is the number of data outputs this bundle has.
+ // OutputCount is the number of data or timer outputs this bundle has.
// We need to see this many closed data channels before the bundle is complete.
OutputCount int
// dataWait is how we determine if a bundle is finished, by waiting for each of
Review Comment:
```suggestion
// DataWait is how we determine if a bundle is finished, by waiting for each of
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org