You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "changliiu (via GitHub)" <gi...@apache.org> on 2025/05/23 22:13:26 UTC

[PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

changliiu opened a new pull request, #35042:
URL: https://github.com/apache/beam/pull/35042

   To improve the performance of the query for GetUnfinishedMinWatermark, we can add a filter to the query.
   Note that the query run on a cache to refresh every 1s. We can remember the cached value every time and use the value as a filter in the next time query.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2962460425

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @chamikaramj for label java.
   R: @nielm for label spanner.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2114498501


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -42,13 +42,23 @@ public class AsyncWatermarkCache implements WatermarkCache {
   private static final Object MIN_WATERMARK_KEY = new Object();
   private final LoadingCache<Object, Optional<Timestamp>> cache;
 
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
+
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =
         CacheBuilder.newBuilder()
             .refreshAfterWrite(java.time.Duration.ofMillis(refreshRate.getMillis()))
             .build(
                 CacheLoader.asyncReloading(
-                    CacheLoader.from(key -> Optional.ofNullable(dao.getUnfinishedMinWatermark())),
+                    CacheLoader.from(
+                        key -> {
+                          Timestamp unfinishedMinTimes =

Review Comment:
   Done. Make the cache value nullable variable instead of optional. Since the cache returns nullable anyway, we don't need to convert nullable -> optional -> nullable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2145546823


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -40,23 +39,40 @@ public class AsyncWatermarkCache implements WatermarkCache {
 
   private static final String THREAD_NAME_FORMAT = "watermark_loading_thread_%d";
   private static final Object MIN_WATERMARK_KEY = new Object();
-  private final LoadingCache<Object, Optional<Timestamp>> cache;
+  private final LoadingCache<Object, Timestamp> cache;
+
+  // This is to cache the result of getUnfinishedMinWatermark query and filter the query in the next
+  // run. For the initial query, the value of this cache is min timestamp. If there is no partition
+  // in the metadata table, then this cache will not be updated. If the getUnfinishedMinWatermark
+  // query fails or times out, then this cache will not be updated.
+  // Note that, all the reload operations on this key are serialized due to use of the single
+  // threaded async reloading executor.
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
 
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =
         CacheBuilder.newBuilder()
             .refreshAfterWrite(java.time.Duration.ofMillis(refreshRate.getMillis()))
             .build(
                 CacheLoader.asyncReloading(
-                    CacheLoader.from(key -> Optional.ofNullable(dao.getUnfinishedMinWatermark())),
+                    CacheLoader.from(
+                        key -> {
+                          Timestamp unfinishedMinTimes =
+                              dao.getUnfinishedMinWatermarkFrom(lastCachedMinWatermark);
+                          if (unfinishedMinTimes != null
+                              && lastCachedMinWatermark.compareTo(unfinishedMinTimes) < 0) {
+                            lastCachedMinWatermark = unfinishedMinTimes;

Review Comment:
   done.
   
   However I'm not sure I understand it correctly, is there more than one thread can execute this cache reload logic?
   
   Resolve as this question is not a blocker.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2114535620


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/SpannerChangeStreamErrorTest.java:
##########
@@ -460,19 +460,27 @@ private void mockGetPartitionsAfter(Timestamp timestamp, ResultSet getPartitionR
   }
 
   private void mockGetWatermark(Timestamp watermark) {
+    final String minWatermark = "min_watermark";

Review Comment:
   I only edit one test case because:
   1. Regarding verifying correctness, I run integration test, and also real load against a local build connector, I print out this cache variable and I can see it is updated successfully.
   2. Regarding unit-test coverage. This PR has two parts - query change, and lastCachedMinWatermark change. The former can be covered by this test. The later actually is implementation details. According to [go/tott/708](https://goto.google.com/tott/708), I think this can be considered implementation detailed change and cannot be properly captured by unit-tests.
   
   Please re-open if you have strong opinions here. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2970984677

   Friendly ping :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2150473203


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -40,23 +39,40 @@ public class AsyncWatermarkCache implements WatermarkCache {
 
   private static final String THREAD_NAME_FORMAT = "watermark_loading_thread_%d";
   private static final Object MIN_WATERMARK_KEY = new Object();
-  private final LoadingCache<Object, Optional<Timestamp>> cache;
+  private final LoadingCache<Object, Timestamp> cache;
+
+  // This is to cache the result of getUnfinishedMinWatermark query and filter the query in the next
+  // run. For the initial query, the value of this cache is min timestamp. If there is no partition
+  // in the metadata table, then this cache will not be updated. If the getUnfinishedMinWatermark
+  // query fails or times out, then this cache will not be updated.
+  // Note that, all the reload operations on this key are serialized due to use of the single
+  // threaded async reloading executor.
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
 
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =
         CacheBuilder.newBuilder()
             .refreshAfterWrite(java.time.Duration.ofMillis(refreshRate.getMillis()))
             .build(
                 CacheLoader.asyncReloading(
-                    CacheLoader.from(key -> Optional.ofNullable(dao.getUnfinishedMinWatermark())),
+                    CacheLoader.from(
+                        key -> {
+                          Timestamp unfinishedMinTimes =
+                              dao.getUnfinishedMinWatermarkFrom(lastCachedMinWatermark);
+                          if (unfinishedMinTimes != null
+                              && lastCachedMinWatermark.compareTo(unfinishedMinTimes) < 0) {
+                            lastCachedMinWatermark = unfinishedMinTimes;

Review Comment:
   Done. Used atomic instead of locking.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2114497342


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -42,13 +42,23 @@ public class AsyncWatermarkCache implements WatermarkCache {
   private static final Object MIN_WATERMARK_KEY = new Object();
   private final LoadingCache<Object, Optional<Timestamp>> cache;
 
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
+
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =

Review Comment:
   Done. Added comments. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2114869392


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +179,57 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermark(Optional<Timestamp> since) {
+    Timestamp sinceTimestamp = since.orElse(Timestamp.MIN_VALUE);
     Statement statement;
+    final String minWatermark = "min_watermark";
     if (this.isPostgres()) {
       statement =
           Statement.newBuilder(
-                  "SELECT \""
+                  "SELECT min(\""
                       + COLUMN_WATERMARK
-                      + "\" FROM \""
+                      + "\") as "
+                      + minWatermark
+                      + " FROM \""
                       + metadataTableName
                       + "\" WHERE \""
                       + COLUMN_STATE
                       + "\" != $1"
-                      + " ORDER BY \""
+                      + " AND \""
                       + COLUMN_WATERMARK
-                      + "\" ASC LIMIT 1")
+                      + "\" >= $2")
               .bind("p1")
               .to(State.FINISHED.name())
+              .bind("p2")
+              .to(sinceTimestamp)
               .build();
     } else {
       statement =
           Statement.newBuilder(
-                  "SELECT "
+                  "SELECT min("

Review Comment:
   There is a discussion to replace "limit 1" using "min()" in order to better use the index. I agree that two queries can have the same result. Please re-open if you have strong opinion on this. Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2905935685

   Integration test can pass locally https://screenshot.googleplex.com/6aQBPAW8YT69VTd


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #35042:
URL: https://github.com/apache/beam/pull/35042


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2955621799

   Reminder, please take a look at this pr: @m-trieu @nielm 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "thiagotnunes (via GitHub)" <gi...@apache.org>.
thiagotnunes commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2144237814


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +178,56 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermarkFrom(Timestamp sinceTimestamp) {
     Statement statement;
+    final String minWatermark = "min_watermark";
     if (this.isPostgres()) {
       statement =
           Statement.newBuilder(
-                  "SELECT \""
+                  "SELECT min(\""

Review Comment:
   nit: uppercase MIN



##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/SpannerChangeStreamErrorTest.java:
##########
@@ -460,19 +460,27 @@ private void mockGetPartitionsAfter(Timestamp timestamp, ResultSet getPartitionR
   }
 
   private void mockGetWatermark(Timestamp watermark) {
+    final String minWatermark = "min_watermark";
+    // The query needs to sync with getUnfinishedMinWatermark() in PartitionMetadataDao file.
     Statement watermarkStatement =
         Statement.newBuilder(
-                "SELECT Watermark FROM my-metadata-table WHERE State != @state ORDER BY Watermark ASC LIMIT 1")
+                "SELECT min(Watermark) as "

Review Comment:
   nit: uppercase MIN



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/NoOpWatermarkCache.java:
##########
@@ -35,6 +35,6 @@ public NoOpWatermarkCache(PartitionMetadataDao dao) {
 
   @Override
   public @Nullable Timestamp getUnfinishedMinWatermark() {
-    return dao.getUnfinishedMinWatermark();
+    return dao.getUnfinishedMinWatermarkFrom(Timestamp.MIN_VALUE);

Review Comment:
   Is the reasoning of having MIN_VALUE here to disable the filtering if we face any issue?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +178,56 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermarkFrom(Timestamp sinceTimestamp) {
     Statement statement;
+    final String minWatermark = "min_watermark";
     if (this.isPostgres()) {
       statement =
           Statement.newBuilder(
-                  "SELECT \""
+                  "SELECT min(\""
                       + COLUMN_WATERMARK
-                      + "\" FROM \""
+                      + "\") as "
+                      + minWatermark
+                      + " FROM \""
                       + metadataTableName
                       + "\" WHERE \""
                       + COLUMN_STATE
                       + "\" != $1"
-                      + " ORDER BY \""
+                      + " AND \""
                       + COLUMN_WATERMARK
-                      + "\" ASC LIMIT 1")
+                      + "\" >= $2")
               .bind("p1")
               .to(State.FINISHED.name())
+              .bind("p2")
+              .to(sinceTimestamp)
               .build();
     } else {
       statement =
           Statement.newBuilder(
-                  "SELECT "
+                  "SELECT min("

Review Comment:
   nit: uppercase MIN



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "dedocibula (via GitHub)" <gi...@apache.org>.
dedocibula commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2120954833


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +179,57 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermark(Optional<Timestamp> since) {

Review Comment:
   Hmm how about getUnfinishedMinWatermarkFrom(Timestamp since). Just finding:
   
   dao.getUnfinishedMinWatermarkFrom(Timestamp.MIN_VALUE);
   
   more readable than
   
   dao.getUnfinishedMinWatermark(Optional.empty());



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2906263499

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2167389124


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -42,13 +42,23 @@ public class AsyncWatermarkCache implements WatermarkCache {
   private static final Object MIN_WATERMARK_KEY = new Object();
   private final LoadingCache<Object, Optional<Timestamp>> cache;
 
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
+
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =
         CacheBuilder.newBuilder()
             .refreshAfterWrite(java.time.Duration.ofMillis(refreshRate.getMillis()))
             .build(
                 CacheLoader.asyncReloading(
-                    CacheLoader.from(key -> Optional.ofNullable(dao.getUnfinishedMinWatermark())),
+                    CacheLoader.from(
+                        key -> {
+                          Timestamp unfinishedMinTimes =

Review Comment:
   I think this is a fairly small issue. 
   Now:
   1. the dao.getUnfinishedMinWatermarkFrom return null. 
   2. The CacheLoader checks do checks and return optional.ofnullable. 
   3. DetectNewPartitionsAction check null or not.
   4. note that dao.getUnfinishedMinWatermarkFrom is also used by NoOpWatermarkCache.
   
   If we return optional by dao.getUnfinishedMinWatermarkFrom, then all the above code needs to change (time-consuming and error-prone), the benefit is only reducing one conversion from null -> optional.
   
   Hence I lean toward keep this code as is.
   I can reply so. But do let me know your thoughts on this if you disagree then we can chat. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2143297879


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +179,57 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermark(Optional<Timestamp> since) {

Review Comment:
   SG, done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2967689469

   Friendly ping :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2913381240

   Run integration test locally based on earlier commit https://github.com/apache/beam/pull/34883 and it passed. Hence the integration timeout is not caused by this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2913386012

   Assigning reviewers:
   
   R: @m-trieu for label java.
   R: @nielm for label spanner.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2913382602

   assign set of reviewers


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "dedocibula (via GitHub)" <gi...@apache.org>.
dedocibula commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2167352769


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -42,13 +42,23 @@ public class AsyncWatermarkCache implements WatermarkCache {
   private static final Object MIN_WATERMARK_KEY = new Object();
   private final LoadingCache<Object, Optional<Timestamp>> cache;
 
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
+
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =
         CacheBuilder.newBuilder()
             .refreshAfterWrite(java.time.Duration.ofMillis(refreshRate.getMillis()))
             .build(
                 CacheLoader.asyncReloading(
-                    CacheLoader.from(key -> Optional.ofNullable(dao.getUnfinishedMinWatermark())),
+                    CacheLoader.from(
+                        key -> {
+                          Timestamp unfinishedMinTimes =

Review Comment:
   Reviving this thread, can we wrap the result as optional at the Timestamp unfinishedMinTimes assignment place? It should shorten the if clause as you can use ifPresent to set this lastCachedMinWatermark field



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "dedocibula (via GitHub)" <gi...@apache.org>.
dedocibula commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2111482619


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +179,57 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermark(Optional<Timestamp> since) {

Review Comment:
   Can we make this input timestamp since definitive (non-optional)? Have the caller give the exact timestamp? I think it would be preferable to have timestamp filtering come from a single place



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -42,13 +42,23 @@ public class AsyncWatermarkCache implements WatermarkCache {
   private static final Object MIN_WATERMARK_KEY = new Object();
   private final LoadingCache<Object, Optional<Timestamp>> cache;
 
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
+
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =

Review Comment:
   Can we add a comment here how the the cache loading works? Namely let's cover:
   - what happens when there are no partition data in the metadata table
   - what happens if the call fails - deadline exceeded or spanner exception
   - what happens in the happy case
   
   Let's also call out that all the reload operations on this key are serialized due to use of the single threaded asyncReloadingExecutor



##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/SpannerChangeStreamErrorTest.java:
##########
@@ -460,19 +460,27 @@ private void mockGetPartitionsAfter(Timestamp timestamp, ResultSet getPartitionR
   }
 
   private void mockGetWatermark(Timestamp watermark) {
+    final String minWatermark = "min_watermark";

Review Comment:
   Shall we add more interesting test cases? Throwing exceptions? Returning null? Returning older value?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -42,13 +42,23 @@ public class AsyncWatermarkCache implements WatermarkCache {
   private static final Object MIN_WATERMARK_KEY = new Object();
   private final LoadingCache<Object, Optional<Timestamp>> cache;
 
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
+
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =
         CacheBuilder.newBuilder()
             .refreshAfterWrite(java.time.Duration.ofMillis(refreshRate.getMillis()))
             .build(
                 CacheLoader.asyncReloading(
-                    CacheLoader.from(key -> Optional.ofNullable(dao.getUnfinishedMinWatermark())),
+                    CacheLoader.from(
+                        key -> {
+                          Timestamp unfinishedMinTimes =

Review Comment:
   Shall we wrap this value in nullable here since we have to do this anyway?
   
   Also, we should only assign unfinishedMinTimes to lastCachedMinWatermark only if it is after it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2114393653


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +179,57 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermark(Optional<Timestamp> since) {

Review Comment:
   I tend to make it clear here that this "since" parameter is optional - call sites can omit this field if they don't care, instead of figuring out what does this mean and give a valid value.
   
   Please re-open if you have strong opinion. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2145527835


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/NoOpWatermarkCache.java:
##########
@@ -35,6 +35,6 @@ public NoOpWatermarkCache(PartitionMetadataDao dao) {
 
   @Override
   public @Nullable Timestamp getUnfinishedMinWatermark() {
-    return dao.getUnfinishedMinWatermark();
+    return dao.getUnfinishedMinWatermarkFrom(Timestamp.MIN_VALUE);

Review Comment:
   we change the unfinished min watermark query with a filter. The filter will only work if the async cache is used to update the cache every time the cached value is updated. 
   
   By contrast, the NoOpWatermarkCache will not take advantage of this filter.
   
   For the sake of readability, we adjust the function of query unfinished watermark to `getUnfinishedMinWatermarkFrom`. And in NoOpWatermarkCache we give the parameter as the min timestamp.
   
   Please re-open or we can discuss offline if this doesn't answer your question. Thanks! 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2145524206


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/SpannerChangeStreamErrorTest.java:
##########
@@ -460,19 +460,27 @@ private void mockGetPartitionsAfter(Timestamp timestamp, ResultSet getPartitionR
   }
 
   private void mockGetWatermark(Timestamp watermark) {
+    final String minWatermark = "min_watermark";
+    // The query needs to sync with getUnfinishedMinWatermark() in PartitionMetadataDao file.
     Statement watermarkStatement =
         Statement.newBuilder(
-                "SELECT Watermark FROM my-metadata-table WHERE State != @state ORDER BY Watermark ASC LIMIT 1")
+                "SELECT min(Watermark) as "

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "thiagotnunes (via GitHub)" <gi...@apache.org>.
thiagotnunes commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2146405346


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -40,23 +39,40 @@ public class AsyncWatermarkCache implements WatermarkCache {
 
   private static final String THREAD_NAME_FORMAT = "watermark_loading_thread_%d";
   private static final Object MIN_WATERMARK_KEY = new Object();
-  private final LoadingCache<Object, Optional<Timestamp>> cache;
+  private final LoadingCache<Object, Timestamp> cache;
+
+  // This is to cache the result of getUnfinishedMinWatermark query and filter the query in the next
+  // run. For the initial query, the value of this cache is min timestamp. If there is no partition
+  // in the metadata table, then this cache will not be updated. If the getUnfinishedMinWatermark
+  // query fails or times out, then this cache will not be updated.
+  // Note that, all the reload operations on this key are serialized due to use of the single
+  // threaded async reloading executor.
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
 
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =
         CacheBuilder.newBuilder()
             .refreshAfterWrite(java.time.Duration.ofMillis(refreshRate.getMillis()))
             .build(
                 CacheLoader.asyncReloading(
-                    CacheLoader.from(key -> Optional.ofNullable(dao.getUnfinishedMinWatermark())),
+                    CacheLoader.from(
+                        key -> {
+                          Timestamp unfinishedMinTimes =
+                              dao.getUnfinishedMinWatermarkFrom(lastCachedMinWatermark);
+                          if (unfinishedMinTimes != null
+                              && lastCachedMinWatermark.compareTo(unfinishedMinTimes) < 0) {
+                            lastCachedMinWatermark = unfinishedMinTimes;

Review Comment:
   We initialize the cache as a single thread executor, but in theory this could be executed in multiple threads. Please make the lastCachedMinWatermark atomic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "nancyxu825 (via GitHub)" <gi...@apache.org>.
nancyxu825 commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2114765150


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +179,57 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermark(Optional<Timestamp> since) {
+    Timestamp sinceTimestamp = since.orElse(Timestamp.MIN_VALUE);
     Statement statement;
+    final String minWatermark = "min_watermark";
     if (this.isPostgres()) {
       statement =
           Statement.newBuilder(
-                  "SELECT \""
+                  "SELECT min(\""
                       + COLUMN_WATERMARK
-                      + "\" FROM \""
+                      + "\") as "
+                      + minWatermark
+                      + " FROM \""
                       + metadataTableName
                       + "\" WHERE \""
                       + COLUMN_STATE
                       + "\" != $1"
-                      + " ORDER BY \""
+                      + " AND \""
                       + COLUMN_WATERMARK
-                      + "\" ASC LIMIT 1")
+                      + "\" >= $2")
               .bind("p1")
               .to(State.FINISHED.name())
+              .bind("p2")
+              .to(sinceTimestamp)
               .build();
     } else {
       statement =
           Statement.newBuilder(
-                  "SELECT "
+                  "SELECT min("

Review Comment:
   Why did you add the changes to add "min" to the query?
   
   Technically, you can still do: SELECT COLUMN_WATERMARK from metadatatable where state != FINISHED and Watermark >= previous watermark ORDER BY COLUMN_WATERMARK ASC LIMIT 1?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2977391370

   Friendly ping :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #35042:
URL: https://github.com/apache/beam/pull/35042#issuecomment-2985728503

   It seems GCP IO integration test stucks indefinitely. Retriggered the test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "thiagotnunes (via GitHub)" <gi...@apache.org>.
thiagotnunes commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2144247256


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/cache/AsyncWatermarkCache.java:
##########
@@ -40,23 +39,40 @@ public class AsyncWatermarkCache implements WatermarkCache {
 
   private static final String THREAD_NAME_FORMAT = "watermark_loading_thread_%d";
   private static final Object MIN_WATERMARK_KEY = new Object();
-  private final LoadingCache<Object, Optional<Timestamp>> cache;
+  private final LoadingCache<Object, Timestamp> cache;
+
+  // This is to cache the result of getUnfinishedMinWatermark query and filter the query in the next
+  // run. For the initial query, the value of this cache is min timestamp. If there is no partition
+  // in the metadata table, then this cache will not be updated. If the getUnfinishedMinWatermark
+  // query fails or times out, then this cache will not be updated.
+  // Note that, all the reload operations on this key are serialized due to use of the single
+  // threaded async reloading executor.
+  private Timestamp lastCachedMinWatermark = Timestamp.MIN_VALUE;
 
   public AsyncWatermarkCache(PartitionMetadataDao dao, Duration refreshRate) {
     this.cache =
         CacheBuilder.newBuilder()
             .refreshAfterWrite(java.time.Duration.ofMillis(refreshRate.getMillis()))
             .build(
                 CacheLoader.asyncReloading(
-                    CacheLoader.from(key -> Optional.ofNullable(dao.getUnfinishedMinWatermark())),
+                    CacheLoader.from(
+                        key -> {
+                          Timestamp unfinishedMinTimes =
+                              dao.getUnfinishedMinWatermarkFrom(lastCachedMinWatermark);
+                          if (unfinishedMinTimes != null
+                              && lastCachedMinWatermark.compareTo(unfinishedMinTimes) < 0) {
+                            lastCachedMinWatermark = unfinishedMinTimes;

Review Comment:
   lastCachedMinWatermark needs to be thread-safe.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add filter to GetMinUnfinishedWatermark queryAdd filter tester [beam]

Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on code in PR #35042:
URL: https://github.com/apache/beam/pull/35042#discussion_r2145523910


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +178,56 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermarkFrom(Timestamp sinceTimestamp) {
     Statement statement;
+    final String minWatermark = "min_watermark";
     if (this.isPostgres()) {
       statement =
           Statement.newBuilder(
-                  "SELECT \""
+                  "SELECT min(\""

Review Comment:
   done



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java:
##########
@@ -178,47 +178,56 @@ public List<String> findAllTableIndexes() {
    *
    * @return the earliest partition watermark which is not in a {@link State#FINISHED} state.
    */
-  public @Nullable Timestamp getUnfinishedMinWatermark() {
+  public @Nullable Timestamp getUnfinishedMinWatermarkFrom(Timestamp sinceTimestamp) {
     Statement statement;
+    final String minWatermark = "min_watermark";
     if (this.isPostgres()) {
       statement =
           Statement.newBuilder(
-                  "SELECT \""
+                  "SELECT min(\""
                       + COLUMN_WATERMARK
-                      + "\" FROM \""
+                      + "\") as "
+                      + minWatermark
+                      + " FROM \""
                       + metadataTableName
                       + "\" WHERE \""
                       + COLUMN_STATE
                       + "\" != $1"
-                      + " ORDER BY \""
+                      + " AND \""
                       + COLUMN_WATERMARK
-                      + "\" ASC LIMIT 1")
+                      + "\" >= $2")
               .bind("p1")
               .to(State.FINISHED.name())
+              .bind("p2")
+              .to(sinceTimestamp)
               .build();
     } else {
       statement =
           Statement.newBuilder(
-                  "SELECT "
+                  "SELECT min("

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org