You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "shunping (via GitHub)" <gi...@apache.org> on 2025/09/13 05:15:21 UTC
[PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]
shunping opened a new pull request, #36142:
URL: https://github.com/apache/beam/pull/36142
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3301251131
Moving it to draft for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3301250669
Got some failure while running the java validatesRunner tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36142:
URL: https://github.com/apache/beam/pull/36142#discussion_r2346926149
##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -892,70 +892,72 @@ func (em *ElementManager) PersistBundle(rb RunBundle, col2Coders map[string]PCol
// Clear out the inprogress elements associated with the completed bundle.
// Must be done after adding the new pending elements to avoid an incorrect
// watermark advancement.
- stage.mu.Lock()
- completed := stage.inprogress[rb.BundleID]
- em.addPending(-len(completed.es))
- delete(stage.inprogress, rb.BundleID)
- for k := range stage.inprogressKeysByBundle[rb.BundleID] {
- delete(stage.inprogressKeys, k)
- }
- delete(stage.inprogressKeysByBundle, rb.BundleID)
-
- // Adjust holds as needed.
- for h, c := range newHolds {
- if c > 0 {
- stage.watermarkHolds.Add(h, c)
- } else if c < 0 {
- stage.watermarkHolds.Drop(h, -c)
+ func() {
Review Comment:
Move everything before `em.markChangedAndClearBundle` into an anonymous function and call defer lock in it. In this case, even if there is any exception or panic happens in this code (e.g. waitgroup negative count due to a some mistake), the runner won't be deadlocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping closed pull request #36142: [Prism] Avoid firing on-time pane for some triggers.
URL: https://github.com/apache/beam/pull/36142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36142:
URL: https://github.com/apache/beam/pull/36142#discussion_r2346926149
##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -892,70 +892,72 @@ func (em *ElementManager) PersistBundle(rb RunBundle, col2Coders map[string]PCol
// Clear out the inprogress elements associated with the completed bundle.
// Must be done after adding the new pending elements to avoid an incorrect
// watermark advancement.
- stage.mu.Lock()
- completed := stage.inprogress[rb.BundleID]
- em.addPending(-len(completed.es))
- delete(stage.inprogress, rb.BundleID)
- for k := range stage.inprogressKeysByBundle[rb.BundleID] {
- delete(stage.inprogressKeys, k)
- }
- delete(stage.inprogressKeysByBundle, rb.BundleID)
-
- // Adjust holds as needed.
- for h, c := range newHolds {
- if c > 0 {
- stage.watermarkHolds.Add(h, c)
- } else if c < 0 {
- stage.watermarkHolds.Drop(h, -c)
+ func() {
Review Comment:
Move everything before `em.markChangedAndClearBundle` into an anonymous function and call defer in it. In this case, even if there is any exception or panic happens in this code (e.g. waitgroup negative count due to a some mistake), the runner won't be deadlocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3300876837
Assigning reviewers:
R: @lostluck for label go.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36142:
URL: https://github.com/apache/beam/pull/36142#discussion_r2353974166
##########
sdks/go/test/integration/primitives/windowinto.go:
##########
@@ -214,7 +214,7 @@ func TriggerElementCount(s beam.Scope) {
validateCount(s.Scope("Fixed"), window.NewFixedWindows(windowSize), col,
[]beam.WindowIntoOption{
beam.Trigger(trigger.AfterCount(2)),
- }, 2)
+ }, 1)
Review Comment:
The count here is one, because the output only contains one aggregated element for window [0, 10000], which is the sum of 1.0 and 2.0.
- `AfterCount` is going to be repeatedly applied.
- No on-time pane fired at the end of window.
- For window [50000, 60000], there is one one element so the trigger is not fired either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3328129290
Reminder, please take a look at this pr: @lostluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3330891575
Looks like the extra pane is supposed to fire because of the closing behavior.
Closing this PR as it is longer relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3289503536
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/36142?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `79.87013%` with `31 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.87%. Comparing base ([`fe5c288`](https://app.codecov.io/gh/apache/beam/commit/fe5c2888ae43678d2498b07e588c0037502220a2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7653a13`](https://app.codecov.io/gh/apache/beam/commit/7653a13b20a210c27a921677f986a2e2b75071c7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/36142?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...am/runners/prism/internal/engine/elementmanager.go](https://app.codecov.io/gh/apache/beam/pull/36142?src=pr&el=tree&filepath=sdks%2Fgo%2Fpkg%2Fbeam%2Frunners%2Fprism%2Finternal%2Fengine%2Felementmanager.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ByaXNtL2ludGVybmFsL2VuZ2luZS9lbGVtZW50bWFuYWdlci5nbw==) | 72.83% | [18 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/beam/pull/36142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pkg/beam/runners/prism/internal/engine/strategy.go](https://app.codecov.io/gh/apache/beam/pull/36142?src=pr&el=tree&filepath=sdks%2Fgo%2Fpkg%2Fbeam%2Frunners%2Fprism%2Finternal%2Fengine%2Fstrategy.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ByaXNtL2ludGVybmFsL2VuZ2luZS9zdHJhdGVneS5nbw==) | 87.67% | [6 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/beam/pull/36142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #36142 +/- ##
============================================
+ Coverage 56.85% 56.87% +0.01%
Complexity 3385 3385
============================================
Files 1220 1220
Lines 185467 185518 +51
Branches 3520 3520
============================================
+ Hits 105450 105509 +59
+ Misses 76687 76672 -15
- Partials 3330 3337 +7
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/36142/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [go](https://app.codecov.io/gh/apache/beam/pull/36142/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `28.27% <79.87%> (+0.05%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/36142?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36142:
URL: https://github.com/apache/beam/pull/36142#discussion_r2346926149
##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -892,70 +892,72 @@ func (em *ElementManager) PersistBundle(rb RunBundle, col2Coders map[string]PCol
// Clear out the inprogress elements associated with the completed bundle.
// Must be done after adding the new pending elements to avoid an incorrect
// watermark advancement.
- stage.mu.Lock()
- completed := stage.inprogress[rb.BundleID]
- em.addPending(-len(completed.es))
- delete(stage.inprogress, rb.BundleID)
- for k := range stage.inprogressKeysByBundle[rb.BundleID] {
- delete(stage.inprogressKeys, k)
- }
- delete(stage.inprogressKeysByBundle, rb.BundleID)
-
- // Adjust holds as needed.
- for h, c := range newHolds {
- if c > 0 {
- stage.watermarkHolds.Add(h, c)
- } else if c < 0 {
- stage.watermarkHolds.Drop(h, -c)
+ func() {
Review Comment:
Move everything before `em.markChangedAndClearBundle` into an anonymous function and call defer lock in it. In this case, even if there is any exception or panic happens in this code (e.g. waitgroup negative count due to a mistake I made earlier), the runner won't be deadlocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org