You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/12/21 23:18:23 UTC

[GitHub] [beam] boyuanzz opened a new pull request #13592: [WIP] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

boyuanzz opened a new pull request #13592:
URL: https://github.com/apache/beam/pull/13592


   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam
 .apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.a
 pache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam
 .apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on a change in pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13592:
URL: https://github.com/apache/beam/pull/13592#discussion_r547445369



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -439,11 +444,37 @@ public IsBounded isBounded() {
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
+    private Cache<UnboundedSourceRestriction, UnboundedReader> cachedReaders;
 
     private UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
     }
 
+    private UnboundedSourceRestriction createCacheKey(
+        UnboundedSource<OutputT, CheckpointT> source, CheckpointT checkpoint) {
+      // For caching reader, we don't care about the watermark.
+      return UnboundedSourceRestriction.create(
+          source, checkpoint, BoundedWindow.TIMESTAMP_MIN_VALUE);
+    }
+
+    @Setup
+    public void setUp() throws Exception {
+      cachedReaders =
+          CacheBuilder.newBuilder()
+              .expireAfterWrite(5, TimeUnit.MINUTES)

Review comment:
       5 mins might be high for DirectRunner but I feel like a few seconds might be too small for a distributed system like Dataflow, especially for a long run streaming application. How about we start from 1min?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749757695


   > > I verified that the reader is reused from cache in Kafka case manually.
   > 
   > Hm, are you sure? That confuses me, because looking at the code, I'm not sure how that could function with `System.identityHashCode` as hashCode for CheckpointMark. Provided that AutoValue delegates its hashCode as would be expected and that guava's Cache uses hashCode. Hm, maybe Dataflow is not using SplittableDoFnViaKeyedWorkItems and has some specific implementation? 
   
   All changes are in SDK side, as well as the cache creation, So it will be independent from runner executions(and I'm using beam_fn_api as well).
   
   But you are right that `hashCode` is not correct if it is not implemented correctly.
   
   > > It makes me feel like configuring split frequency from PipelineOption
   > 
   > Sure, Flink has such an option. It would be natural to either create one for generic use, or add that to respective runner's PipelineOptions.
    
   The checkpoint for SDF is different from Flink checkpoint. That says, even the checkpoint interval is not configured for Flink, the checkpoint for SDF will still happen based on how `OutputAndTimeBoundedSplittableProcessElementInvoker` is configured.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik edited a comment on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik edited a comment on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749512239


   > because it should be impossible for single DoFn instance to read from multiple readers.
   
   Sorry, that is not true. Single instance can process multiple sources (after split). That seems problematic. Maybe we can ignore that, but that will probably cause issues with DirectRunner, because I think it can clone sources to simulate recovery from checkpoint.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz merged pull request #13592: [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz merged pull request #13592:
URL: https://github.com/apache/beam/pull/13592


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749467845


   One more concern - the current implementation relies on `CheckpointMark#hashCode` and `CheckpointMark#equals`. It is likely that these will not have these two correctly implemented. We should stick to `Coder#structuralValue` for that.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749730945


   btw I tested it by using Kafka + Dataflow streaming and I don't notice a performance improvement there. It might be that the cost of creating Kafka connection is really cheap.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on a change in pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13592:
URL: https://github.com/apache/beam/pull/13592#discussion_r547440367



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -439,11 +444,37 @@ public IsBounded isBounded() {
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
+    private Cache<UnboundedSourceRestriction, UnboundedReader> cachedReaders;
 
     private UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
     }
 
+    private UnboundedSourceRestriction createCacheKey(
+        UnboundedSource<OutputT, CheckpointT> source, CheckpointT checkpoint) {
+      // For caching reader, we don't care about the watermark.
+      return UnboundedSourceRestriction.create(
+          source, checkpoint, BoundedWindow.TIMESTAMP_MIN_VALUE);
+    }
+
+    @Setup
+    public void setUp() throws Exception {
+      cachedReaders =
+          CacheBuilder.newBuilder()
+              .expireAfterWrite(5, TimeUnit.MINUTES)
+              .maximumSize(100)
+              .removalListener(
+                  (RemovalListener<UnboundedSourceRestriction, UnboundedReader>)
+                      removalNotification -> {
+                        try {
+                          removalNotification.getValue().close();

Review comment:
       That's a very good point. Thanks for catching this! I should remove the entry from cache when I obtain the reader from the cache to avoid eviction during processing and we will cache the reader after we finish.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-751292295


   Thanks for your quick review!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik edited a comment on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik edited a comment on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749754120


   > I verified that the reader is reused from cache in Kafka case manually.
   
   Hm, are you sure? That confuses me, because looking at the code, I'm not sure how that could function with `System.identityHashCode` as hashCode for CheckpointMark. Provided that AutoValue delegates its hashCode as would be expected and that guava's Cache uses hashCode. Hm, maybe Dataflow is not using SplittableDoFnViaKeyedWorkItems and has some specific implementation? 
   
   All changes are in SDK side, as well as the cache creation, So it will be independent from runner executions(and I'm using beam_fn_api as well).
   
   But you are right that `hashCode` is not correct if it is not implemented correctly.
   
   > It makes me feel like configuring split frequency from PipelineOption
   
   Sure, Flink has such an option. It would be natural to either create one for generic use, or add that to respective runner's PipelineOptions.
   
   The checkpoint for SDF is different from Flink checkpoint. That says, even the checkpoint interval is not configured for Flink, the checkpoint for SDF will still happen based on how `OutputAndTimeBoundedSplittableProcessElementInvoker` is configured.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749721610


   > One more concern - the current implementation relies on `CheckpointMark#hashCode` and `CheckpointMark#equals`. It is likely that these will not have these two correctly implemented. We should stick to `Coder#structuralValue` for that.
   > The same holds true for UnboundedSource#hashCode and equals, we should probably not use the source in the cache key, because it should be impossible for single DoFn instance to read from multiple readers.
   
   I was thinking about using `Coder#structuralValue` but I'm concerning about the additional overhead from encoding. The cacheKey is created very frequently(at least twice per element) and it's not cheap for coder to encode a value. 
   
   As you mentioned, a DoFn instance could process multiple sources especially the source allows initial split(and we cannot assume that CheckpointMark contains the source info, although in most case it does), that's why I decided to use `UnboundedSourceRestriction` to locate a reader. DirectRunner is using timers(InMemoryTimerInternals) and states(in memory as well) to reschedule checkpoints. It should be a reference instead of a deep copy?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-750577460


   Updated with test. @je-ik 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749754120


   > I verified that the reader is reused from cache in Kafka case manually.
   
   Hm, are you sure? That confuses me, because looking at the code, I'm not sure how that could function with `System.identityHashCode` as hashCode for CheckpointMark. Provided that AutoValue delegates its hashCode as would be expected and that guava's Cache uses hashCode. Hm, maybe Dataflow is not using SplittableDoFnViaKeyedWorkItems and has some specific implementation? 
   
   > It makes me feel like configuring split frequency from PipelineOption
   
   Sure, Flink has such an option. It would be natural to either create one for generic use, or add that to respective runner's PipelineOptions.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik edited a comment on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik edited a comment on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749467845


   One more concern - the current implementation relies on `CheckpointMark#hashCode` and `CheckpointMark#equals`. It is likely that these will not have these two correctly implemented. We should stick to `Coder#structuralValue` for that.
   The same holds true for UnboundedSource#hashCode and equals, we should probably not use the source in the cache key, because it should be impossible for single DoFn instance to read from multiple readers.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749754986


   > Hm, maybe Dataflow is not using SplittableDoFnViaKeyedWorkItems and has some specific implementation?
   
   Ah, got that. Dataflow uses cache for state access. So writing to state and reading it back can return the same identical object. But that is runner specific. Flink always serializes access to state.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-751022474


   Run Java_Examples_Dataflow PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik commented on a change in pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik commented on a change in pull request #13592:
URL: https://github.com/apache/beam/pull/13592#discussion_r547153671



##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -488,7 +519,20 @@ public void splitRestriction(
         restrictionTracker(
             @Restriction UnboundedSourceRestriction<OutputT, CheckpointT> restriction,
             PipelineOptions pipelineOptions) {
-      return new UnboundedSourceAsSDFRestrictionTracker(restriction, pipelineOptions);
+      try {
+        UnboundedReader reader =
+            cachedReaders.getIfPresent(
+                createCacheKey(restriction.getSource(), restriction.getCheckpoint()));
+        if (reader == null) {
+          reader =
+              restriction.getSource().createReader(pipelineOptions, restriction.getCheckpoint());
+          cachedReaders.put(
+              createCacheKey(restriction.getSource(), restriction.getCheckpoint()), reader);
+        }
+        return new UnboundedSourceAsSDFRestrictionTracker(restriction, pipelineOptions, reader);

Review comment:
       ```suggestion
           return new UnboundedSourceAsSDFRestrictionTracker<>(restriction, pipelineOptions, reader);
   ```

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -439,11 +444,37 @@ public IsBounded isBounded() {
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
+    private Cache<UnboundedSourceRestriction, UnboundedReader> cachedReaders;

Review comment:
       ```suggestion
       private Cache<UnboundedSourceRestriction<OutputT, CheckointT>, UnboundedReader> cachedReaders;
   ```

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -756,23 +810,20 @@ public CheckpointMark getCheckpointMark() {
       private final PipelineOptions pipelineOptions;
       private UnboundedSource.UnboundedReader<OutputT> currentReader;
       private boolean readerHasBeenStarted;
+      private boolean isDone;

Review comment:
       :+1:

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -439,11 +444,37 @@ public IsBounded isBounded() {
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
+    private Cache<UnboundedSourceRestriction, UnboundedReader> cachedReaders;
 
     private UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
     }
 
+    private UnboundedSourceRestriction createCacheKey(
+        UnboundedSource<OutputT, CheckpointT> source, CheckpointT checkpoint) {
+      // For caching reader, we don't care about the watermark.
+      return UnboundedSourceRestriction.create(
+          source, checkpoint, BoundedWindow.TIMESTAMP_MIN_VALUE);
+    }
+
+    @Setup
+    public void setUp() throws Exception {
+      cachedReaders =
+          CacheBuilder.newBuilder()
+              .expireAfterWrite(5, TimeUnit.MINUTES)

Review comment:
       I think this could be way lower. We insert the reader back to the cache after checkpoint, so if it is not reused after few seconds, it is likely not be reused at all. Maybe 10 seconds might be enough?

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -756,23 +810,20 @@ public CheckpointMark getCheckpointMark() {
       private final PipelineOptions pipelineOptions;
       private UnboundedSource.UnboundedReader<OutputT> currentReader;
       private boolean readerHasBeenStarted;
+      private boolean isDone;
 
       UnboundedSourceAsSDFRestrictionTracker(
           UnboundedSourceRestriction<OutputT, CheckpointT> initialRestriction,
-          PipelineOptions pipelineOptions) {
+          PipelineOptions pipelineOptions,
+          UnboundedReader cachedUnboundedReader) {

Review comment:
       ```suggestion
             UnboundedReader<OutputT> cachedUnboundedReader) {
   ```

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -501,6 +545,10 @@ public ProcessContinuation processElement(
         throws IOException {
       UnboundedSourceRestriction<OutputT, CheckpointT> initialRestriction =
           tracker.currentRestriction();
+      UnboundedReader currentReader =

Review comment:
       ```suggestion
         UnboundedReader<OutputT> currentReader =
   ```

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -527,6 +575,12 @@ public ProcessContinuation processElement(
             currentRestriction.getCheckpoint()::finalizeCheckpoint);
       }
 
+      // Update the readers cache with latest CheckpointMark.
+      cachedReaders.put(
+          createCacheKey(
+              currentReader.getCurrentSource(), (CheckpointT) currentReader.getCheckpointMark()),

Review comment:
       ```suggestion
                 (UnboundedSource<OutputT, CheckpointT>) currentReader.getCurrentSource(),
                 (CheckpointT) currentReader.getCheckpointMark()),
   ```

##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java
##########
@@ -439,11 +444,37 @@ public IsBounded isBounded() {
     private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceAsSDFWrapperFn.class);
     private static final int DEFAULT_BUNDLE_FINALIZATION_LIMIT_MINS = 10;
     private final Coder<CheckpointT> checkpointCoder;
+    private Cache<UnboundedSourceRestriction, UnboundedReader> cachedReaders;
 
     private UnboundedSourceAsSDFWrapperFn(Coder<CheckpointT> checkpointCoder) {
       this.checkpointCoder = checkpointCoder;
     }
 
+    private UnboundedSourceRestriction createCacheKey(
+        UnboundedSource<OutputT, CheckpointT> source, CheckpointT checkpoint) {
+      // For caching reader, we don't care about the watermark.
+      return UnboundedSourceRestriction.create(
+          source, checkpoint, BoundedWindow.TIMESTAMP_MIN_VALUE);
+    }
+
+    @Setup
+    public void setUp() throws Exception {
+      cachedReaders =
+          CacheBuilder.newBuilder()
+              .expireAfterWrite(5, TimeUnit.MINUTES)
+              .maximumSize(100)
+              .removalListener(
+                  (RemovalListener<UnboundedSourceRestriction, UnboundedReader>)
+                      removalNotification -> {
+                        try {
+                          removalNotification.getValue().close();

Review comment:
       This looks dangerous. What if we are still using the reader inside `processElement`? This is likely to be called from different thread, so we need a synchronization there and not close the reader if being used.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749744489


   > btw I tested it by using Kafka + Dataflow streaming and I don't notice a performance improvement there. It might be that the cost of creating Kafka connection is really cheap.
   
   That doesn't seem to be a surpise, because under the current implementation, it is essential for CheckpointMark to correctly implement equals and hashCode (which KafkaCheckpointMark does not), because between two successive calls to `processElement` the checkpoint is stored in state and therefore serialized and deserialized and so a new object is put into the cache. Second point is that, even after we fix this, it will be probably noticeable only on pipelines with very frequent checkpoints.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749735258


   > I was thinking about using `Coder#structuralValue` but I'm concerning about the additional overhead from encoding. The cacheKey is created very frequently(at least twice per element) and it's not cheap for coder to encode a value.
   
   That should not be a problem as in most cases, processing a single `element` in this case means emitting quite many elements downstream. The cost of the structuralValue should be pretty much amortized I would say. We could do something to calculate it only once per element?
   > 
   > As you mentioned, a DoFn instance could process multiple sources especially the source allows initial split(and we cannot assume that CheckpointMark contains the source info, although in most case it does), that's why I decided to use `UnboundedSourceRestriction` to locate a reader. DirectRunner is using timers(InMemoryTimerInternals) and states(in memory as well) to reschedule checkpoints. It should be a reference instead of a deep copy?
   
   I don't think we can use a reference either, because the UnboundedSource is Serializable and any runner is free to clone it (which is what DirectRunner does, afaik). The best solution would seem to be to mark each initially split (via @SplitRestriction) restriction with unique ID and then transfer this ID to all residual restrictions. There should be always be at most one "active" (either currently being processed or having non-null residual) restriction, so we could use that for identifying the reader without referencing Source (which might be problematic, as implementing hashCode and/or equals for UnboundedSource will not be a common practice, I'm afraid).
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] je-ik commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
je-ik commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749512239


   > because it should be impossible for single DoFn instance to read from multiple readers.
   Sorry, that is not true. Single instance can process multiple sources (after split). That seems problematic. Maybe we can ignore that, but that will probably cause issues with DirectRunner, because I think it can clone sources to simulate recovery from checkpoint.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749929365


   I did more changes to address mentioned concerns, which include:
   
   - Have DoFn keep the Cache for all created RestrictionTracker
   - The RestrictionTracker#tryClaim will get the reader from the cache if any.
   - When RestrictionTracker#trySplit happens, the reader will be cached by the latest CheckpointMark.
   
   I'll try to have a test tomorrow.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-750427408


   > Can we add some simple test to verify this in `ReadTest`?
   
   That's my plan. I'll let you know when it's ready.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749748837


   > > btw I tested it by using Kafka + Dataflow streaming and I don't notice a performance improvement there. It might be that the cost of creating Kafka connection is really cheap.
   > 
   > That doesn't seem to be a surpise, because under the current implementation, it is essential for CheckpointMark to correctly implement equals and hashCode (which KafkaCheckpointMark does not), because between two successive calls to `processElement` the checkpoint is stored in state and therefore serialized and deserialized and so a new object is put into the cache. 
   
   I verified that the reader is reused from cache in Kafka case manually. 
   
   > Second point is that, even after we fix this, it will be probably noticeable only on pipelines with very frequent checkpoints.
   
   It makes me feel like configuring split frequency from PipelineOption or Read API still has value. If comparing the way between DirectRunner invokes Unbounded Read and Unbounded SDF, the checkpoint happens every 100 elements for Unbounded Read but almost every second for SDF. Besides, SDF uses timers and states to feed back checkpoint to process, which brings more overhead for processing.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-750427257


   > Great, I think this covers pretty much all the topics. I have only one small concern left - because the key in cache is derived from structuralValue of RestrictionCoder, it uses SerializableCoder for encoding the source. That might not be 100% stable. I would prefer to generate a different identifier in `UnboundedSourceAsSDFWrapperFn.splitRestriction` for each split and then carry it along all calls to `trySplit`. But this might be viewed as minor concern.
   
   I would not worry about that too much since we are using `Source` + `CheckpointMark` as the restriction, as well as the cache key.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on pull request #13592: [WIP] Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #13592:
URL: https://github.com/apache/beam/pull/13592#issuecomment-749313927


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org