You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "scwhittle (via GitHub)" <gi...@apache.org> on 2025/02/06 14:48:10 UTC

[PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

scwhittle opened a new pull request, #33901:
URL: https://github.com/apache/beam/pull/33901

   Add a utility class to enable sharing across all deserialized instances of a DoFn and use it in UnboundedSourceAsSdfWrapperFn to cache Readers across dofn instances.
   
   fixes #32968
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "sjvanrossum (via GitHub)" <gi...@apache.org>.
sjvanrossum commented on code in PR #33901:
URL: https://github.com/apache/beam/pull/33901#discussion_r1952954467


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java:
##########
@@ -481,12 +482,31 @@ static class UnboundedSourceAsSDFWrapperFn<OutputT, CheckpointT extends Checkpoi
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
-    private @Nullable Cache<Object, UnboundedReader<OutputT>> cachedReaders;
+    private final PerSerializationStatic<Cache<Object, UnboundedReader<OutputT>>> cachedReaders;
     private @Nullable Coder<UnboundedSourceRestriction<OutputT, CheckpointT>> restrictionCoder;
 
     @VisibleForTesting
     UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
+      cachedReaders =
+          new PerSerializationStatic<>(
+              () ->
+                  CacheBuilder.newBuilder()
+                      .expireAfterWrite(1, TimeUnit.MINUTES)
+                      .maximumSize(100)

Review Comment:
   Aside from that, the manual cache management implemented in the restriction tracker looks like it could be replaced by applying `#expireAfterAccess(long, TimeUnit)` and `#build(CacheLoader<? super K1,V1>)`, but that's a separate issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #33901:
URL: https://github.com/apache/beam/pull/33901#discussion_r1962402392


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java:
##########
@@ -481,12 +485,38 @@ static class UnboundedSourceAsSDFWrapperFn<OutputT, CheckpointT extends Checkpoi
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
-    private @Nullable Cache<Object, UnboundedReader<OutputT>> cachedReaders;
+    private final MemoizingPerInstantiationSerializableSupplier<
+            Cache<Object, UnboundedReader<OutputT>>>
+        readerCacheSupplier;
+    private static final Executor closeExecutor =
+        Executors.newCachedThreadPool(
+            new ThreadFactoryBuilder().setNameFormat("UnboundedReaderCloses-%d").build());
     private @Nullable Coder<UnboundedSourceRestriction<OutputT, CheckpointT>> restrictionCoder;
 
     @VisibleForTesting
     UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
+      readerCacheSupplier =
+          new MemoizingPerInstantiationSerializableSupplier<>(
+              () ->
+                  CacheBuilder.newBuilder()
+                      .expireAfterWrite(1, TimeUnit.MINUTES)
+                      .removalListener(
+                          (RemovalListener<Object, UnboundedReader<OutputT>>)
+                              removalNotification -> {
+                                if (removalNotification.getCause() != RemovalCause.EXPLICIT) {
+                                  closeExecutor.execute(
+                                      () -> {
+                                        try {
+                                          Preconditions.checkNotNull(removalNotification.getValue())

Review Comment:
   nit: this doesn't actually change anything, as it will just throw an NPE same as if you didn't use it. So prefer to use Beam SDK's own Preconditions which have `checkArgumentNotNull` (throws `IllegalArgumentException`) and `checkStateNotNull` (throws `IllegalStateException`)
   
   Out of curiosity, I looked at the docs on RemovalNotification I believe if the value is null here that means it was already garbage collected. Since you aren't using `weakValues` this will never happen. But if you wanted this code to be agnostic to the cache configuration a bit, you could just check if it is null and return early.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle closed pull request #33901: Change UnboundedSourceAsSdfWrapperFn to share the cache across instances.
URL: https://github.com/apache/beam/pull/33901


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2652050581

   @sjvanrossum I tried to abstract your caching in the KafkaSDF here since there would be similar concerns about a static cache for this dofn.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2681776452

   failures are unrelated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2671245916

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle closed pull request #33901: Change UnboundedSourceAsSdfWrapperFn to share the cache across instances.
URL: https://github.com/apache/beam/pull/33901


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "sjvanrossum (via GitHub)" <gi...@apache.org>.
sjvanrossum commented on code in PR #33901:
URL: https://github.com/apache/beam/pull/33901#discussion_r1952944361


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java:
##########
@@ -481,12 +482,31 @@ static class UnboundedSourceAsSDFWrapperFn<OutputT, CheckpointT extends Checkpoi
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
-    private @Nullable Cache<Object, UnboundedReader<OutputT>> cachedReaders;
+    private final PerSerializationStatic<Cache<Object, UnboundedReader<OutputT>>> cachedReaders;
     private @Nullable Coder<UnboundedSourceRestriction<OutputT, CheckpointT>> restrictionCoder;
 
     @VisibleForTesting
     UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
+      cachedReaders =
+          new PerSerializationStatic<>(
+              () ->
+                  CacheBuilder.newBuilder()
+                      .expireAfterWrite(1, TimeUnit.MINUTES)
+                      .maximumSize(100)

Review Comment:
   The time-based eviction policy seems to be aligned with the reader cache in the Dataflow runner, but the size-based eviction policy is not present in the Dataflow runner. I'm thinking this would negatively impact `Read` instances with > 100 splits.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "sjvanrossum (via GitHub)" <gi...@apache.org>.
sjvanrossum commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2654322616

   > @sjvanrossum I tried to abstract your caching in the KafkaSDF here since there would be similar concerns about a static cache for this dofn.
   
   That looks good to me, thanks for distilling it into a utility class! As to naming, this is essentially a `Supplier<T>` with additional guarantees about the return value of `#get()`.
   
   Drawing inspiration from Guava's [`Suppliers`](https://guava.dev/releases/32.1.2-jre/api/docs/com/google/common/base/Suppliers.html) `#memoize(Supplier<T>)`, `#memoizeWithExpiration(Supplier, long, TimeUnit)` and `#ofInstance​(T)` backed by `MemoizingSupplier`, `ExpiringMemoizingSupplier` and `SupplierOfInstance` respectively, I think `SerializableSupplierMemoizingPerInstantiation<T>` conveys the intended application of this class.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle closed pull request #33901: Change UnboundedSourceAsSdfWrapperFn to share the cache across instances.
URL: https://github.com/apache/beam/pull/33901


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #33901:
URL: https://github.com/apache/beam/pull/33901#discussion_r1954364366


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java:
##########
@@ -481,12 +482,31 @@ static class UnboundedSourceAsSDFWrapperFn<OutputT, CheckpointT extends Checkpoi
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
-    private @Nullable Cache<Object, UnboundedReader<OutputT>> cachedReaders;
+    private final PerSerializationStatic<Cache<Object, UnboundedReader<OutputT>>> cachedReaders;
     private @Nullable Coder<UnboundedSourceRestriction<OutputT, CheckpointT>> restrictionCoder;
 
     @VisibleForTesting
     UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
+      cachedReaders =
+          new PerSerializationStatic<>(
+              () ->
+                  CacheBuilder.newBuilder()
+                      .expireAfterWrite(1, TimeUnit.MINUTES)
+                      .maximumSize(100)

Review Comment:
   Thanks! I didn't was just copying but you're correct that this seems like it introduce  a problem with the wider scope of the cache.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle merged PR #33901:
URL: https://github.com/apache/beam/pull/33901


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2652047977

   assign set of reviewers
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2671220457

   Run Java_IOs_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2681465952

   Run Java_Pulsar_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2652052142

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @m-trieu for label java.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "sjvanrossum (via GitHub)" <gi...@apache.org>.
sjvanrossum commented on code in PR #33901:
URL: https://github.com/apache/beam/pull/33901#discussion_r1952954467


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java:
##########
@@ -481,12 +482,31 @@ static class UnboundedSourceAsSDFWrapperFn<OutputT, CheckpointT extends Checkpoi
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
-    private @Nullable Cache<Object, UnboundedReader<OutputT>> cachedReaders;
+    private final PerSerializationStatic<Cache<Object, UnboundedReader<OutputT>>> cachedReaders;
     private @Nullable Coder<UnboundedSourceRestriction<OutputT, CheckpointT>> restrictionCoder;
 
     @VisibleForTesting
     UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
+      cachedReaders =
+          new PerSerializationStatic<>(
+              () ->
+                  CacheBuilder.newBuilder()
+                      .expireAfterWrite(1, TimeUnit.MINUTES)
+                      .maximumSize(100)

Review Comment:
   Aside from that, the manual cache management implemented in the restriction tracker looks like it could be replaced by applying `#expireAfterAccess(long, TimeUnit)` and `#build(CacheLoader)`, but that's a separate issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2669096816

   @kennknowles Could you take another look? I realized that we have to worry now about concurrent cache access (before the DoFn being single-use-at-a-time prevented that) and the get/invalidate as separate methods could be racy. So I looked at [dataflow runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderCache.java](https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderCache.java) and changed to use the concurrentmap directly as it does. While there I noticed that there were previously problems due to close blocking so I made this similarly async. http://issues.apache.org/jira/browse/BEAM-11401


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2681466478

   Run Java_IOs_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2665753579

   Run Java_IOs_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2665753112

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #33901:
URL: https://github.com/apache/beam/pull/33901#discussion_r1960771308


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/util/MemoizingPerInstantiationSerializableSupplier.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.annotation.Nullable;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.NonNull;
+
+/**
+ * A supplier that memoizes within an instantiation across serialization/deserialization.
+ *
+ * <p>Specifically the wrapped supplier will be called once and the result memoized per group consisting
+ * of an instance and all instances deserialized from its serialized state.
+ *
+ * <p>A particular use for this is within a DoFn class to maintain shared state across all instances

Review Comment:
   Fancy! It took me a minute to understand how this works.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2670850733

   > Interesting. The race is still there, right? The async close could be racing with some other usage of it, no? But it won't block the current thread is the main thing.
   
   The race was more with the get from the cache and separate invalidate. Since there are simulataneous dofns both could get the same value where we want more of acquire/checkout semantics so that there is just one user.  That was addressed using the atomic asMap().remove() instead of the separate get/invalidate methods.
   
   It's true long closes could still be an issue but at least they aren't blocking unrelated processing threads.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2640042173

   R: @kennknowles
   
   I don't really like the utility class name, let me know if you have some better idea.  It seems like this could be a common use within DoFns. For example, @sjvanrossum made me think of this concern in his work on Kafka source in #32986


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2640045657

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2665754160

   Run Java_Pulsar_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Change UnboundedSourceAsSdfWrapperFn to share the cache across instances. [beam]

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33901:
URL: https://github.com/apache/beam/pull/33901#issuecomment-2681468499

   Run Java_Amazon-Web-Services2_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org