You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "shunping (via GitHub)" <gi...@apache.org> on 2025/09/13 05:15:21 UTC

[PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]

shunping opened a new pull request, #36142:
URL: https://github.com/apache/beam/pull/36142

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3301251131

   Moving it to draft for now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3301250669

   Got some failure while running the java validatesRunner tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36142:
URL: https://github.com/apache/beam/pull/36142#discussion_r2346926149


##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -892,70 +892,72 @@ func (em *ElementManager) PersistBundle(rb RunBundle, col2Coders map[string]PCol
 	// Clear out the inprogress elements associated with the completed bundle.
 	// Must be done after adding the new pending elements to avoid an incorrect
 	// watermark advancement.
-	stage.mu.Lock()
-	completed := stage.inprogress[rb.BundleID]
-	em.addPending(-len(completed.es))
-	delete(stage.inprogress, rb.BundleID)
-	for k := range stage.inprogressKeysByBundle[rb.BundleID] {
-		delete(stage.inprogressKeys, k)
-	}
-	delete(stage.inprogressKeysByBundle, rb.BundleID)
-
-	// Adjust holds as needed.
-	for h, c := range newHolds {
-		if c > 0 {
-			stage.watermarkHolds.Add(h, c)
-		} else if c < 0 {
-			stage.watermarkHolds.Drop(h, -c)
+	func() {

Review Comment:
   Move everything before `em.markChangedAndClearBundle` into an anonymous function and call defer lock in it. In this case, even if there is any exception or panic happens in this code (e.g. waitgroup negative count due to a some mistake), the runner won't be deadlocked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping closed pull request #36142: [Prism] Avoid firing on-time pane for some triggers.
URL: https://github.com/apache/beam/pull/36142


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36142:
URL: https://github.com/apache/beam/pull/36142#discussion_r2346926149


##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -892,70 +892,72 @@ func (em *ElementManager) PersistBundle(rb RunBundle, col2Coders map[string]PCol
 	// Clear out the inprogress elements associated with the completed bundle.
 	// Must be done after adding the new pending elements to avoid an incorrect
 	// watermark advancement.
-	stage.mu.Lock()
-	completed := stage.inprogress[rb.BundleID]
-	em.addPending(-len(completed.es))
-	delete(stage.inprogress, rb.BundleID)
-	for k := range stage.inprogressKeysByBundle[rb.BundleID] {
-		delete(stage.inprogressKeys, k)
-	}
-	delete(stage.inprogressKeysByBundle, rb.BundleID)
-
-	// Adjust holds as needed.
-	for h, c := range newHolds {
-		if c > 0 {
-			stage.watermarkHolds.Add(h, c)
-		} else if c < 0 {
-			stage.watermarkHolds.Drop(h, -c)
+	func() {

Review Comment:
   Move everything before `em.markChangedAndClearBundle` into an anonymous function and call defer in it. In this case, even if there is any exception or panic happens in this code (e.g. waitgroup negative count due to a some mistake), the runner won't be deadlocked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3300876837

   Assigning reviewers:
   
   R: @lostluck for label go.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36142:
URL: https://github.com/apache/beam/pull/36142#discussion_r2353974166


##########
sdks/go/test/integration/primitives/windowinto.go:
##########
@@ -214,7 +214,7 @@ func TriggerElementCount(s beam.Scope) {
 	validateCount(s.Scope("Fixed"), window.NewFixedWindows(windowSize), col,
 		[]beam.WindowIntoOption{
 			beam.Trigger(trigger.AfterCount(2)),
-		}, 2)
+		}, 1)

Review Comment:
   The count here is one, because the output only contains one aggregated element for window [0, 10000], which is the sum of 1.0 and 2.0. 
   - `AfterCount` is going to be repeatedly applied. 
   - No on-time pane fired at the end of window.
   - For window [50000, 60000], there is one one element so the trigger is not fired either.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3328129290

   Reminder, please take a look at this pr: @lostluck 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Prism] Avoid firing on-time pane for some triggers. [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3330891575

   Looks like the extra pane is supposed to fire because of the closing behavior.
   
   Closing this PR as it is longer relevant.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #36142:
URL: https://github.com/apache/beam/pull/36142#issuecomment-3289503536

   ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/36142?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `79.87013%` with `31 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 56.87%. Comparing base ([`fe5c288`](https://app.codecov.io/gh/apache/beam/commit/fe5c2888ae43678d2498b07e588c0037502220a2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7653a13`](https://app.codecov.io/gh/apache/beam/commit/7653a13b20a210c27a921677f986a2e2b75071c7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 2 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/36142?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...am/runners/prism/internal/engine/elementmanager.go](https://app.codecov.io/gh/apache/beam/pull/36142?src=pr&el=tree&filepath=sdks%2Fgo%2Fpkg%2Fbeam%2Frunners%2Fprism%2Finternal%2Fengine%2Felementmanager.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ByaXNtL2ludGVybmFsL2VuZ2luZS9lbGVtZW50bWFuYWdlci5nbw==) | 72.83% | [18 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/beam/pull/36142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...pkg/beam/runners/prism/internal/engine/strategy.go](https://app.codecov.io/gh/apache/beam/pull/36142?src=pr&el=tree&filepath=sdks%2Fgo%2Fpkg%2Fbeam%2Frunners%2Fprism%2Finternal%2Fengine%2Fstrategy.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ByaXNtL2ludGVybmFsL2VuZ2luZS9zdHJhdGVneS5nbw==) | 87.67% | [6 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/beam/pull/36142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #36142      +/-   ##
   ============================================
   + Coverage     56.85%   56.87%   +0.01%     
     Complexity     3385     3385              
   ============================================
     Files          1220     1220              
     Lines        185467   185518      +51     
     Branches       3520     3520              
   ============================================
   + Hits         105450   105509      +59     
   + Misses        76687    76672      -15     
   - Partials       3330     3337       +7     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/beam/pull/36142/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [go](https://app.codecov.io/gh/apache/beam/pull/36142/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `28.27% <79.87%> (+0.05%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/36142?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix extra on-time pane fired at non-after-watermark triggers. [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36142:
URL: https://github.com/apache/beam/pull/36142#discussion_r2346926149


##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -892,70 +892,72 @@ func (em *ElementManager) PersistBundle(rb RunBundle, col2Coders map[string]PCol
 	// Clear out the inprogress elements associated with the completed bundle.
 	// Must be done after adding the new pending elements to avoid an incorrect
 	// watermark advancement.
-	stage.mu.Lock()
-	completed := stage.inprogress[rb.BundleID]
-	em.addPending(-len(completed.es))
-	delete(stage.inprogress, rb.BundleID)
-	for k := range stage.inprogressKeysByBundle[rb.BundleID] {
-		delete(stage.inprogressKeys, k)
-	}
-	delete(stage.inprogressKeysByBundle, rb.BundleID)
-
-	// Adjust holds as needed.
-	for h, c := range newHolds {
-		if c > 0 {
-			stage.watermarkHolds.Add(h, c)
-		} else if c < 0 {
-			stage.watermarkHolds.Drop(h, -c)
+	func() {

Review Comment:
   Move everything before `em.markChangedAndClearBundle` into an anonymous function and call defer lock in it. In this case, even if there is any exception or panic happens in this code (e.g. waitgroup negative count due to a mistake I made earlier), the runner won't be deadlocked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org