You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tonytanger (via GitHub)" <gi...@apache.org> on 2023/07/31 16:59:10 UTC

[GitHub] [beam] tonytanger opened a new pull request, #27755: Remove reconciled partitions from list of missing partitions

tonytanger opened a new pull request, #27755:
URL: https://github.com/apache/beam/pull/27755

   After missing partitions have been reconciled and outputted, remove them from the list of missing partitions. This prevents a race condition that leads to multiple reconcile runs consecutively. Added an unit test to verify this works.
   
   Also increase the wait time before reconciling because typically reconciler is only needed in pipelines that are slow to process partitions. Adding extra wait time gives the pipeline time to catch up.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tonytanger commented on a diff in pull request #27755: Remove reconciled partitions from list of missing partitions

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on code in PR #27755:
URL: https://github.com/apache/beam/pull/27755#discussion_r1283377400


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/reconciler/PartitionReconciler.java:
##########
@@ -100,6 +101,7 @@ public PartitionReconciler(MetadataTableDao metadataTableDao, ChangeStreamMetric
    * @param missingPartitions partitions not being streamed.
    */
   public void addMissingPartitions(List<ByteStringRange> missingPartitions) {
+    hasAddedMissingPartitions = true;

Review Comment:
   It should not be necessary to reset the flag. Though there's no downside to resetting it.
   
   The purpose of this flag is to ensure we don't call `getPartitionsToReconcile` before it. If we do call `getPartitionsToReconcile` before `addMissingPartitions` we run into the problem of  We only call `getPartitionsToReconcile` once per run and the entire `PartitionReconciler` is reset at the start of every run.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tonytanger commented on a diff in pull request #27755: Remove reconciled partitions from list of missing partitions

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on code in PR #27755:
URL: https://github.com/apache/beam/pull/27755#discussion_r1283378423


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java:
##########
@@ -325,26 +328,26 @@ public ProcessContinuation run(
       Optional<Instant> maybeWatermark =
           getNewWatermark(streamPartitionsWithWatermark, newPartitions);
       maybeWatermark.ifPresent(metadataTableDao::updateDetectNewPartitionWatermark);
-      // Using NewPartitions and StreamPartitions, evaluate partitions that are possibly not being
-      // streamed. This isn't perfect because there may be partitions moving between
-      // StreamPartitions and NewPartitions while scanning the metadata table. Also, this does not
-      // include NewPartitions marked as deleted from a previous DNP run not yet processed by RCSP.
-      List<ByteStringRange> existingPartitions =
-          streamPartitionsWithWatermark.stream()
-              .map(StreamPartitionWithWatermark::getPartition)
-              .collect(Collectors.toList());
-      existingPartitions.addAll(outputtedNewPartitions);
-      List<ByteStringRange> missingStreamPartitions =
-          getMissingPartitionsFromEntireKeySpace(existingPartitions);
-      orphanedMetadataCleaner.addMissingPartitions(missingStreamPartitions);
-      partitionReconciler.addMissingPartitions(missingStreamPartitions);
-    }
-
-    // Only start reconciling after the pipeline has been running for a while.
-    if (tracker.currentRestriction().getFrom() > 50) {
-      processReconcilerPartitions(
-          receiver, watermarkEstimator, initialPipelineState.getStartTime());
-      cleanUpOrphanedMetadata();
+      // Only start reconciling after the pipeline has been running for a while.
+      if (tracker.currentRestriction().getFrom() > 50) {

Review Comment:
   InitialRestriction is set to 0 and it increments once per processElement. The purpose is to allow the partitions outputted by DetectNewPartitions to be processed by the next step ReadChangeStreamPartition. Often times there could be thousands to hundreds of thousands of partitions outputted. We just want to allow some time for them to start being processed by ReadChangeStreamPartition. This reduces the noise that reconciler can create during the initial starting period of a pipeline. This is more of a precaution than a strict requirement.
   
   We haven't tested precisely in terms what 50 means in terms of real time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on a diff in pull request #27755: Remove reconciled partitions from list of missing partitions

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #27755:
URL: https://github.com/apache/beam/pull/27755#discussion_r1283331908


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/reconciler/PartitionReconciler.java:
##########
@@ -100,6 +101,7 @@ public PartitionReconciler(MetadataTableDao metadataTableDao, ChangeStreamMetric
    * @param missingPartitions partitions not being streamed.
    */
   public void addMissingPartitions(List<ByteStringRange> missingPartitions) {
+    hasAddedMissingPartitions = true;

Review Comment:
   I see this controls getPartitionsToReconcile's behavior. Currently once set to true it never turns back. Should this flag be reset after partitionsToReconcile are handled correctly somewhere?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java:
##########
@@ -325,26 +328,26 @@ public ProcessContinuation run(
       Optional<Instant> maybeWatermark =
           getNewWatermark(streamPartitionsWithWatermark, newPartitions);
       maybeWatermark.ifPresent(metadataTableDao::updateDetectNewPartitionWatermark);
-      // Using NewPartitions and StreamPartitions, evaluate partitions that are possibly not being
-      // streamed. This isn't perfect because there may be partitions moving between
-      // StreamPartitions and NewPartitions while scanning the metadata table. Also, this does not
-      // include NewPartitions marked as deleted from a previous DNP run not yet processed by RCSP.
-      List<ByteStringRange> existingPartitions =
-          streamPartitionsWithWatermark.stream()
-              .map(StreamPartitionWithWatermark::getPartition)
-              .collect(Collectors.toList());
-      existingPartitions.addAll(outputtedNewPartitions);
-      List<ByteStringRange> missingStreamPartitions =
-          getMissingPartitionsFromEntireKeySpace(existingPartitions);
-      orphanedMetadataCleaner.addMissingPartitions(missingStreamPartitions);
-      partitionReconciler.addMissingPartitions(missingStreamPartitions);
-    }
-
-    // Only start reconciling after the pipeline has been running for a while.
-    if (tracker.currentRestriction().getFrom() > 50) {
-      processReconcilerPartitions(
-          receiver, watermarkEstimator, initialPipelineState.getStartTime());
-      cleanUpOrphanedMetadata();
+      // Only start reconciling after the pipeline has been running for a while.
+      if (tracker.currentRestriction().getFrom() > 50) {

Review Comment:
   This assumes restriction incremented to 50 means "pipeline has been running for a while". iirc there is no guaranteed correspondence to the restriction (long number) to real time In general. What is the purpose here, and is it tested how robust this condition would be?
   
   CC: @johnjcasey for SDF expertise.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn merged pull request #27755: Remove reconciled partitions from list of missing partitions

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #27755:
URL: https://github.com/apache/beam/pull/27755


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #27755: Remove reconciled partitions from list of missing partitions

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27755:
URL: https://github.com/apache/beam/pull/27755#issuecomment-1658901136

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @kennknowles for label java.
   R: @manavgarg for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tonytanger commented on pull request #27755: Remove reconciled partitions from list of missing partitions

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on PR #27755:
URL: https://github.com/apache/beam/pull/27755#issuecomment-1658933288

   @jackdingilian has reviewed this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org