You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/08/24 17:57:51 UTC
[GitHub] [beam] damccorm opened a new pull request, #22846: WIP: combining state integration test
damccorm opened a new pull request, #22846:
URL: https://github.com/apache/beam/pull/22846
Dont merge
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22846: WIP: combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1226055137
Run Go PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22846:
URL: https://github.com/apache/beam/pull/22846#discussion_r954396578
##########
sdks/go/test/integration/primitives/state.go:
##########
@@ -126,3 +132,120 @@ func BagStateParDo() *beam.Pipeline {
return p
}
+
+type combiningStateFn struct {
+ State0 state.Combining[int, int, int]
+ State1 state.Combining[int, int, int]
+ State2 state.Combining[string, string, int]
+ State3 state.Combining[string, string, int]
+ State4 state.Combining[int, int, int]
+}
+
+type accum1 struct{}
+
+func (ac *accum1) MergeAccumulators(a, b int) int {
+ return a + b
+}
+
+type accum2 struct{}
Review Comment:
Yeah, that's fair - I updated to combine*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
lostluck commented on code in PR #22846:
URL: https://github.com/apache/beam/pull/22846#discussion_r954340508
##########
sdks/go/test/integration/primitives/state.go:
##########
@@ -126,3 +132,120 @@ func BagStateParDo() *beam.Pipeline {
return p
}
+
+type combiningStateFn struct {
+ State0 state.Combining[int, int, int]
+ State1 state.Combining[int, int, int]
+ State2 state.Combining[string, string, int]
+ State3 state.Combining[string, string, int]
+ State4 state.Combining[int, int, int]
+}
+
+type accum1 struct{}
+
+func (ac *accum1) MergeAccumulators(a, b int) int {
+ return a + b
+}
+
+type accum2 struct{}
Review Comment:
accum* isn't a good name for these structs since they're combineFns and not accumulators. comb* would be better as prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1226128887
# [Codecov](https://codecov.io/gh/apache/beam/pull/22846?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#22846](https://codecov.io/gh/apache/beam/pull/22846?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c8c360d) into [master](https://codecov.io/gh/apache/beam/commit/4d19439380bcba7032e0e0f8004e42d101d37f76?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4d19439) will **decrease** coverage by `0.05%`.
> The diff coverage is `40.80%`.
> :exclamation: Current head c8c360d differs from pull request most recent head 66132cc. Consider uploading reports for the commit 66132cc to get more accurate results
```diff
@@ Coverage Diff @@
## master #22846 +/- ##
==========================================
- Coverage 73.93% 73.88% -0.06%
==========================================
Files 713 713
Lines 94151 94318 +167
==========================================
+ Hits 69610 69683 +73
- Misses 23256 23347 +91
- Partials 1285 1288 +3
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `51.18% <40.80%> (-0.06%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/22846?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/graph/fn.go](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2ZuLmdv) | `84.54% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/exec/translate.go](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy90cmFuc2xhdGUuZ28=) | `13.20% <0.00%> (-0.35%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/exec/userstate.go](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy91c2Vyc3RhdGUuZ28=) | `11.04% <0.00%> (-1.78%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/graphx/translate.go](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyYW5zbGF0ZS5nbw==) | `39.83% <0.00%> (-1.24%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/state/state.go](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3N0YXRlL3N0YXRlLmdv) | `75.00% <74.73%> (-0.31%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.01% <0.00%> (-1.39%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.09% <0.00%> (+0.15%)` | :arrow_up: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `93.43% <0.00%> (+0.38%)` | :arrow_up: |
| [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `94.02% <0.00%> (+0.74%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/apache/beam/pull/22846/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1227322636
Run Go PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a diff in pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
riteshghorse commented on code in PR #22846:
URL: https://github.com/apache/beam/pull/22846#discussion_r955006450
##########
sdks/go/test/integration/primitives/state.go:
##########
@@ -126,3 +132,120 @@ func BagStateParDo() *beam.Pipeline {
return p
}
+
+type combiningStateFn struct {
+ State0 state.Combining[int, int, int]
+ State1 state.Combining[int, int, int]
+ State2 state.Combining[string, string, int]
+ State3 state.Combining[string, string, int]
+ State4 state.Combining[int, int, int]
+}
+
+type combine1 struct{}
+
+func (ac *combine1) MergeAccumulators(a, b int) int {
+ return a + b
+}
+
+type combine2 struct{}
+
+func (ac *combine2) MergeAccumulators(a, b string) string {
+ ai, _ := strconv.Atoi(a)
+ bi, _ := strconv.Atoi(b)
+ return strconv.Itoa(ai + bi)
+}
+
+func (ac *combine2) ExtractOutput(a string) int {
+ ai, _ := strconv.Atoi(a)
+ return ai
+}
+
+type combine3 struct{}
+
+func (ac *combine3) CreateAccumulator() string {
+ return "0"
+}
+
+func (ac *combine3) MergeAccumulators(a string, b string) string {
+ ai, _ := strconv.Atoi(a)
+ bi, _ := strconv.Atoi(b)
+ return strconv.Itoa(ai + bi)
+}
+
+func (ac *combine3) ExtractOutput(a string) int {
+ ai, _ := strconv.Atoi(a)
+ return ai
+}
+
+type combine4 struct{}
+
+func (ac *combine4) AddInput(a, b int) int {
+ return a + b
+}
+
+func (ac *combine4) MergeAccumulators(a, b int) int {
+ return a + b
+}
+
+func (f *combiningStateFn) ProcessElement(s state.Provider, w string, c int) string {
+ i, _, err := f.State0.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ err = f.State0.Add(s, 1)
+ if err != nil {
+ panic(err)
+ }
+ i1, _, err := f.State1.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ err = f.State1.Add(s, 1)
+ if err != nil {
+ panic(err)
+ }
+ i2, _, err := f.State2.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ err = f.State2.Add(s, "1")
+ if err != nil {
+ panic(err)
+ }
+ i3, _, err := f.State3.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ err = f.State3.Add(s, "1")
+ if err != nil {
+ panic(err)
+ }
+ i4, _, err := f.State4.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ f.State4.Add(s, 1)
Review Comment:
missing error handling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1226284501
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm merged pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm merged PR #22846:
URL: https://github.com/apache/beam/pull/22846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22846:
URL: https://github.com/apache/beam/pull/22846#discussion_r955018245
##########
sdks/go/test/integration/primitives/state.go:
##########
@@ -126,3 +132,120 @@ func BagStateParDo() *beam.Pipeline {
return p
}
+
+type combiningStateFn struct {
+ State0 state.Combining[int, int, int]
+ State1 state.Combining[int, int, int]
+ State2 state.Combining[string, string, int]
+ State3 state.Combining[string, string, int]
+ State4 state.Combining[int, int, int]
+}
+
+type combine1 struct{}
+
+func (ac *combine1) MergeAccumulators(a, b int) int {
+ return a + b
+}
+
+type combine2 struct{}
+
+func (ac *combine2) MergeAccumulators(a, b string) string {
+ ai, _ := strconv.Atoi(a)
+ bi, _ := strconv.Atoi(b)
+ return strconv.Itoa(ai + bi)
+}
+
+func (ac *combine2) ExtractOutput(a string) int {
+ ai, _ := strconv.Atoi(a)
+ return ai
+}
+
+type combine3 struct{}
+
+func (ac *combine3) CreateAccumulator() string {
+ return "0"
+}
+
+func (ac *combine3) MergeAccumulators(a string, b string) string {
+ ai, _ := strconv.Atoi(a)
+ bi, _ := strconv.Atoi(b)
+ return strconv.Itoa(ai + bi)
+}
+
+func (ac *combine3) ExtractOutput(a string) int {
+ ai, _ := strconv.Atoi(a)
+ return ai
+}
+
+type combine4 struct{}
+
+func (ac *combine4) AddInput(a, b int) int {
+ return a + b
+}
+
+func (ac *combine4) MergeAccumulators(a, b int) int {
+ return a + b
+}
+
+func (f *combiningStateFn) ProcessElement(s state.Provider, w string, c int) string {
+ i, _, err := f.State0.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ err = f.State0.Add(s, 1)
+ if err != nil {
+ panic(err)
+ }
+ i1, _, err := f.State1.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ err = f.State1.Add(s, 1)
+ if err != nil {
+ panic(err)
+ }
+ i2, _, err := f.State2.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ err = f.State2.Add(s, "1")
+ if err != nil {
+ panic(err)
+ }
+ i3, _, err := f.State3.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ err = f.State3.Add(s, "1")
+ if err != nil {
+ panic(err)
+ }
+ i4, _, err := f.State4.Read(s)
+ if err != nil {
+ panic(err)
+ }
+ f.State4.Add(s, 1)
Review Comment:
Good catch - fixed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1226089181
Run Go PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1227153842
Run Go Postcommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1226254451
Run Go PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1226261314
R: @riteshghorse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22846: Combining state integration test
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22846:
URL: https://github.com/apache/beam/pull/22846#issuecomment-1226623238
Run Go PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org