You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/08/05 18:41:42 UTC

[GitHub] [beam] dpcollins-google opened a new pull request, #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

dpcollins-google opened a new pull request, #22612:
URL: https://github.com/apache/beam/pull/22612

   SDF is not well supported by the default mode of dataflow, and UnboundedSource is supported by all modes.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r939777332


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {
+      SequencedMessage peeked = subscriber.peek().get();
+      Offset nextOffset = Offset.of(peeked.getCursor().getOffset() + 1);
+      checkState(nextOffset.value() > fetchOffset.value());
+      fetchOffset = nextOffset;
+      lastMessageTimestamp = Optional.of(getTimestamp(peeked));

Review Comment:
   It depends on what the watermark and checkpointMark should reflect- whether they should reflect the current message or all previous messages. If it is all previous messages, the behavior here should be correct: the watermark and checkpoint mark reflect all messages processed in the past. If it should reflect the current message as well, this behavior should be updated to return getCurrentTimestamp for getWatermark and getCurrent().offset() + 1 for the checkpointMark



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209796861

   Run PostCommit_Java_DataflowV2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1210592444

   > Pls fix spotless.
   > 
   > @@ -18,7 +18,6 @@ 17:22:48 package·org.apache.beam.sdk.io.gcp.pubsublite.internal; 17:22:48 17:22:48 import·static·org.junit.Assert.assertEquals; 17:22:48 -import·static·org.junit.Assert.assertThrows; 17:22:48 import·static·org.mockito.Mockito.verify;
   
   Done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r939774784


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();

Review Comment:
   This is the first line of advance(), so this check is already made.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1206754139

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209817686

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] joey-berman commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
joey-berman commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1211323831

   Run Gradle Publish


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209724014

   This failure might be related ? 
   https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Phrase/2/testReport/junit/org.apache.beam.sdk.io.gcp.pubsublite/ReadWriteIT/testReadWrite/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209951492

   Run PostCommit_Java_DataflowV2
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] codecov[bot] commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1207545747

   # [Codecov](https://codecov.io/gh/apache/beam/pull/22612?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#22612](https://codecov.io/gh/apache/beam/pull/22612?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (17fb9c0) into [master](https://codecov.io/gh/apache/beam/commit/54b0784da7ccba738deff22bd83fbc374ad21d2e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (54b0784) will **increase** coverage by `0.01%`.
   > The diff coverage is `63.15%`.
   
   > :exclamation: Current head 17fb9c0 differs from pull request most recent head 2f08b0e. Consider uploading reports for the commit 2f08b0e to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #22612      +/-   ##
   ==========================================
   + Coverage   74.19%   74.20%   +0.01%     
   ==========================================
     Files         706      708       +2     
     Lines       93229    93462     +233     
   ==========================================
   + Hits        69168    69354     +186     
   - Misses      22793    22833      +40     
   - Partials     1268     1275       +7     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | go | `51.52% <56.97%> (+0.02%)` | :arrow_up: |
   | python | `83.61% <72.56%> (+0.04%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/22612?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/go/pkg/beam/core/runtime/exec/input.go](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9pbnB1dC5nbw==) | `51.96% <ø> (+0.99%)` | :arrow_up: |
   | [sdks/go/pkg/beam/runners/direct/direct.go](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RpcmVjdC9kaXJlY3QuZ28=) | `65.76% <0.00%> (-0.60%)` | :arrow_down: |
   | [sdks/go/pkg/beam/validate.go](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS92YWxpZGF0ZS5nbw==) | `42.10% <0.00%> (ø)` | |
   | [...on/apache\_beam/examples/complete/juliaset/setup.py](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvc2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
   | [...m/examples/inference/pytorch\_image\_segmentation.py](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3B5dG9yY2hfaW1hZ2Vfc2VnbWVudGF0aW9uLnB5) | `0.00% <ø> (ø)` | |
   | [...ython/apache\_beam/examples/kafkataxi/kafka\_taxi.py](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMva2Fma2F0YXhpL2thZmthX3RheGkucHk=) | `0.00% <ø> (ø)` | |
   | [sdks/python/apache\_beam/examples/sql\_taxi.py](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc3FsX3RheGkucHk=) | `0.00% <ø> (ø)` | |
   | [...python/apache\_beam/examples/wordcount\_xlang\_sql.py](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X3hsYW5nX3NxbC5weQ==) | `0.00% <ø> (ø)` | |
   | [sdks/python/apache\_beam/io/azure/blobstorageio.py](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXp1cmUvYmxvYnN0b3JhZ2Vpby5weQ==) | `26.88% <0.00%> (ø)` | |
   | [...thon/apache\_beam/ml/inference/pytorch\_inference.py](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL3B5dG9yY2hfaW5mZXJlbmNlLnB5) | `0.00% <0.00%> (ø)` | |
   | ... and [31 more](https://codecov.io/gh/apache/beam/pull/22612/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: Codecov can now indicate which changes are the most critical in Pull Requests. [Learn more](https://about.codecov.io/product/feature/runtime-insights/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1210592059

   Run PostCommit_Java_DataflowV2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r941393326


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/BlockingCommitterImpl.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static com.google.cloud.pubsublite.internal.ExtractStatus.toCanonical;
+
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.internal.wire.Committer;
+import java.util.concurrent.TimeUnit;
+
+public class BlockingCommitterImpl implements BlockingCommitter {
+
+  private final Committer committer;
+
+  BlockingCommitterImpl(Committer committer) {
+    if (!committer.isRunning()) {
+      throw new RuntimeException(committer.failureCause());

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209951651

   Run PostCommit_Java_Dataflow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209453817

   Run PostCommit_Java_DataflowV2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r942007685


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/CheckpointMarkImpl.java:
##########
@@ -62,7 +66,11 @@ public CheckpointMarkImpl decode(InputStream inStream) throws IOException {
 
   @Override
   public void finalizeCheckpoint() {
-    checkState(committer.isPresent());
-    committer.get().commitOffset(offset);
+    try {
+      checkState(committer.isPresent());

Review Comment:
   finalizeCheckpoint call is best effort but does the current implementation rely on this being invoked exactly ?
   For example, will the reader read duplicate messages if a finalizeCheckpoint fails ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r942377949


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/CheckpointMarkImpl.java:
##########
@@ -62,7 +66,11 @@ public CheckpointMarkImpl decode(InputStream inStream) throws IOException {
 
   @Override
   public void finalizeCheckpoint() {
-    checkState(committer.isPresent());
-    committer.get().commitOffset(offset);
+    try {
+      checkState(committer.isPresent());

Review Comment:
   The implementation does not rely on this. It only affects stackdriver metrics and resumption after drain.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r939738263


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {
+      SequencedMessage peeked = subscriber.peek().get();
+      Offset nextOffset = Offset.of(peeked.getCursor().getOffset() + 1);
+      checkState(nextOffset.value() > fetchOffset.value());
+      fetchOffset = nextOffset;
+      lastMessageTimestamp = Optional.of(getTimestamp(peeked));

Review Comment:
   Seems like lastMessageTimestamp and fetchOffset will not be set for the first record ?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();

Review Comment:
   Probably check subscriber state and fail if not State.RUNNING ? Otherwise if the reader doesn't get to the correct state here (State.RUNNING) the reader will never be able to recover. 



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/SubscribeTransform.java:
##########
@@ -134,19 +160,28 @@ private TopicPath getTopicPath() {
     }
   }
 
-  @Override
-  public PCollection<SequencedMessage> expand(PBegin input) {
-    PCollection<SubscriptionPartition> subscriptionPartitions;
-    subscriptionPartitions =
+  @SuppressWarnings("unused")
+  private PCollection<SequencedMessage> expandSdf(PBegin input) {

Review Comment:
   Instead of this, we should introduce a TransformOverride that overrides the SDF-based source transform by the UnboundedSource-based source transform. We do exactly that for Kafka: 
   
   https://github.com/apache/beam/blob/17fb9c0342064cd4375b0d7f2c37e12a175d03ef/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L537
   
   https://github.com/apache/beam/blob/67e6726ffeb47d2ada0122369fa230833ce0f026/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1418
   
   This way Dataflow Runner v2 and other runners will continue to use SDF by default. I think a direct SDF implementation should be more powerful when it comes to capabilities (and potentially more performant) over a SDF wrapped UnboundedSource implementation. The downside is that we'll have to maintain two PSL source implementations but I think this is OK given that the goal is to fully migrate to the SDF version in the future while supporting the current Dataflow production runner well using an UnboundedSource-based source implementation.
   
   WDYT ?
   
   



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {

Review Comment:
   Is the subscriber already set to the first record the first time the start() is called ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r939774567


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/SubscribeTransform.java:
##########
@@ -134,19 +160,28 @@ private TopicPath getTopicPath() {
     }
   }
 
-  @Override
-  public PCollection<SequencedMessage> expand(PBegin input) {
-    PCollection<SubscriptionPartition> subscriptionPartitions;
-    subscriptionPartitions =
+  @SuppressWarnings("unused")
+  private PCollection<SequencedMessage> expandSdf(PBegin input) {

Review Comment:
   I'm not planning on deleting the SDF based implementation, but this seems like a lot of restructuring effort for ambiguous value. All implementations should support UnboundedSource at present IIUC (either directly or via wrapping for runnerv2)
   
   > potentially more performant
   
   The amount of work actually performed by either the SDF or UnboundedSource implementation is so miniscule (pulling out of an in-memory buffer) that I think this is probably moot. I'd note that it does not appear to perform better in testing.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1206753276

   R: @chamikaramj 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209487620

   Thanks Sam. I think we can merge this when tests pass. LGTM.
   
   Based on an offline discussion with Daniel, we'll be adding a transform override in a follow-up PR to keep SDF version the default while overriding with UnboundedSource-based version for production (Runner v1) Dataflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209796731

   Run PostCommit_Java_Dataflow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1210592157

   Run PostCommit_Java_Dataflow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209724528

   Run PostCommit_Java_Dataflow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1210147739

   Pls fix spotless.
   
     @@ -18,7 +18,6 @@
   17:22:48            package·org.apache.beam.sdk.io.gcp.pubsublite.internal;
   17:22:48            
   17:22:48            import·static·org.junit.Assert.assertEquals;
   17:22:48           -import·static·org.junit.Assert.assertThrows;
   17:22:48            import·static·org.mockito.Mockito.verify;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r941387264


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedSourceImpl.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.Partition;
+import com.google.cloud.pubsublite.PartitionLookupUtils;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.extensions.protobuf.ProtoCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.gcp.pubsublite.SubscriberOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class UnboundedSourceImpl extends UnboundedSource<SequencedMessage, CheckpointMarkImpl> {
+
+  private final SubscriberOptions subscriberOptions;
+  private final SubscriberFactory subscriberFactory;
+  private final BacklogReaderFactory readerFactory;
+  private final Optional<Partition> partition;
+
+  interface SubscriberFactory extends Serializable {
+
+    MemoryBufferedSubscriber create(SubscriptionPartition subscriptionPartition, Offset offset);
+  }
+
+  interface BacklogReaderFactory extends Serializable {
+
+    TopicBacklogReader create(SubscriptionPartition subscriptionPartition);
+  }
+
+  UnboundedSourceImpl(
+      SubscriberOptions subscriberOptions,
+      SubscriberFactory subscriberFactory,
+      BacklogReaderFactory readerFactory) {
+    this.subscriberOptions = subscriberOptions;
+    this.subscriberFactory = subscriberFactory;
+    this.readerFactory = readerFactory;
+    this.partition = Optional.absent();
+  }
+
+  private UnboundedSourceImpl(
+      SubscriberOptions subscriberOptions,
+      SubscriberFactory subscriberFactory,
+      BacklogReaderFactory readerFactory,
+      Partition partition) {
+    this.subscriberOptions = subscriberOptions;
+    this.subscriberFactory = subscriberFactory;
+    this.readerFactory = readerFactory;
+    this.partition = Optional.of(partition);
+  }
+
+  @Override
+  public List<? extends UnboundedSource<SequencedMessage, CheckpointMarkImpl>> split(
+      int desiredNumSplits, PipelineOptions options) throws Exception {
+    checkState(!partition.isPresent());
+    int numPartitions = PartitionLookupUtils.numPartitions(subscriberOptions.subscriptionPath());

Review Comment:
   Not really- that would require the user to pay a crazy amount of money.
   
   per:
   >  Would we want multiple partitions to a single source shard
   
   This adds a lot of complexity to the watermarking and commit process, and the low performance risk is mitigated by minimum spends on a per-partition basis in Pub/Sub Lite.
   
   If this ever becomes an issue for a customer, they will be using enough Pub/Sub Lite resources that I will personally go in and add the complexity.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r940543440


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/SubscribeTransform.java:
##########
@@ -134,19 +160,28 @@ private TopicPath getTopicPath() {
     }
   }
 
-  @Override
-  public PCollection<SequencedMessage> expand(PBegin input) {
-    PCollection<SubscriptionPartition> subscriptionPartitions;
-    subscriptionPartitions =
+  @SuppressWarnings("unused")
+  private PCollection<SequencedMessage> expandSdf(PBegin input) {

Review Comment:
   Per offline discussion: this will happen in a child once this PR is merged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r939776054


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {

Review Comment:
   It is if the record is ready- this seems to be the correct behavior and match KafkaUnboundedReader?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r939790974


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {
+      SequencedMessage peeked = subscriber.peek().get();
+      Offset nextOffset = Offset.of(peeked.getCursor().getOffset() + 1);
+      checkState(nextOffset.value() > fetchOffset.value());
+      fetchOffset = nextOffset;
+      lastMessageTimestamp = Optional.of(getTimestamp(peeked));

Review Comment:
   Fixed to match documented semantics in UnboundedReader interface: `An element is considered 'read' if it could be returned by a call to {@link #getCurrent}`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1208281692

   Probably someone from Dataflow streaming team should take a quick look as well. @scwhittle will you be able to check ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] scwhittle commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
scwhittle commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r941358993


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/SubscribeTransform.java:
##########
@@ -101,6 +91,42 @@ private MemoryBufferedSubscriber newBufferedSubscriber(
         consumer -> newSubscriber(subscriptionPartition.partition(), startOffset, consumer));
   }
 
+  private MemoryBufferedSubscriber getCachedSubscriber(
+      SubscriptionPartition subscriptionPartition, Offset startOffset) {
+    Supplier<MemoryBufferedSubscriber> getOrCreate =
+        () ->
+            PerServerSubscriberCache.CACHE.get(
+                subscriptionPartition,
+                () -> newBufferedSubscriber(subscriptionPartition, startOffset));
+    while (true) {
+      MemoryBufferedSubscriber subscriber = getOrCreate.get();
+      Offset fetchOffset = subscriber.fetchOffset();
+      if (startOffset.equals(fetchOffset)) {
+        return subscriber;
+      }
+      LOG.info(
+          "Discarding subscriber due to mismatch, this should be rare. {}, start: {} fetch: {}",
+          subscriptionPartition,
+          startOffset,
+          fetchOffset);
+      try {
+        subscriber.stopAsync().awaitTerminated();

Review Comment:
   does the get above remove it from the cache? Or do you have to do so here so a new one is returned next time?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/SubscribeTransform.java:
##########
@@ -134,19 +160,28 @@ private TopicPath getTopicPath() {
     }
   }
 
-  @Override
-  public PCollection<SequencedMessage> expand(PBegin input) {
-    PCollection<SubscriptionPartition> subscriptionPartitions;
-    subscriptionPartitions =
+  @SuppressWarnings("unused")
+  private PCollection<SequencedMessage> expandSdf(PBegin input) {
+    PCollection<SubscriptionPartition> subscriptionPartitions =
         input.apply(new SubscriptionPartitionLoader(getTopicPath(), options.subscriptionPath()));
-
     return subscriptionPartitions.apply(
         ParDo.of(
             new PerSubscriptionPartitionSdf(
-                new ManagedBacklogReaderFactoryImpl(this::newBacklogReader),
+                new ManagedFactoryImpl<>(this::newBacklogReader),
+                new ManagedFactoryImpl<>(this::newCommitter),
                 this::newInitialOffsetReader,
                 this::newRestrictionTracker,
-                this::newPartitionProcessor,
-                this::newCommitter)));
+                this::newPartitionProcessor)));
+  }
+
+  private PCollection<SequencedMessage> expandSource(PBegin input) {
+    return input.apply(
+        Read.from(
+            new UnboundedSourceImpl(options, this::newBufferedSubscriber, this::newBacklogReader)));
+  }
+
+  @Override
+  public PCollection<SequencedMessage> expand(PBegin input) {
+    return expandSource(input);

Review Comment:
   option to use SDF or Source? Or can SDF be used elsewhere by configuration?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ApiServices.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import com.google.api.core.ApiService;
+import java.util.concurrent.TimeUnit;
+
+class ApiServices {
+
+  private ApiServices() {}
+
+  static AutoCloseable asCloseable(ApiService service) {
+    return () -> service.stopAsync().awaitTerminated(1, TimeUnit.MINUTES);

Review Comment:
   Is exception thrown if this is timed out? Should 1 minute be adjustable?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {
+      subscriber.pop();
+    }
+    Optional<SequencedMessage> next = subscriber.peek();
+    advanced = next.isPresent();
+    if (!advanced) {
+      return false;
+    }
+    Offset nextOffset = Offset.of(next.get().getCursor().getOffset() + 1);
+    checkState(nextOffset.value() > fetchOffset.value());
+    fetchOffset = nextOffset;
+    lastMessageTimestamp = Optional.of(getTimestamp(next.get()));
+    return true;
+  }
+
+  @Override
+  public Instant getWatermark() {
+    return lastMessageTimestamp.orElse(BoundedWindow.TIMESTAMP_MIN_VALUE);

Review Comment:
   are timestamps increasing in PubsubLite?  Or does this need to be based upon last messages and some controllable skew option etc



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/BlockingCommitterImpl.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static com.google.cloud.pubsublite.internal.ExtractStatus.toCanonical;
+
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.internal.wire.Committer;
+import java.util.concurrent.TimeUnit;
+
+public class BlockingCommitterImpl implements BlockingCommitter {
+
+  private final Committer committer;
+
+  BlockingCommitterImpl(Committer committer) {
+    if (!committer.isRunning()) {
+      throw new RuntimeException(committer.failureCause());
+    }
+    this.committer = committer;
+  }
+
+  @Override
+  public void commitOffset(Offset offset) {
+    try {
+      committer.commitOffset(offset).get(1, TimeUnit.MINUTES);

Review Comment:
   ditto should this timeout come from option?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedSourceImpl.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.Partition;
+import com.google.cloud.pubsublite.PartitionLookupUtils;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.extensions.protobuf.ProtoCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.gcp.pubsublite.SubscriberOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class UnboundedSourceImpl extends UnboundedSource<SequencedMessage, CheckpointMarkImpl> {
+
+  private final SubscriberOptions subscriberOptions;
+  private final SubscriberFactory subscriberFactory;
+  private final BacklogReaderFactory readerFactory;
+  private final Optional<Partition> partition;
+
+  interface SubscriberFactory extends Serializable {
+
+    MemoryBufferedSubscriber create(SubscriptionPartition subscriptionPartition, Offset offset);
+  }
+
+  interface BacklogReaderFactory extends Serializable {
+
+    TopicBacklogReader create(SubscriptionPartition subscriptionPartition);
+  }
+
+  UnboundedSourceImpl(
+      SubscriberOptions subscriberOptions,
+      SubscriberFactory subscriberFactory,
+      BacklogReaderFactory readerFactory) {
+    this.subscriberOptions = subscriberOptions;
+    this.subscriberFactory = subscriberFactory;
+    this.readerFactory = readerFactory;
+    this.partition = Optional.absent();
+  }
+
+  private UnboundedSourceImpl(
+      SubscriberOptions subscriberOptions,
+      SubscriberFactory subscriberFactory,
+      BacklogReaderFactory readerFactory,
+      Partition partition) {
+    this.subscriberOptions = subscriberOptions;
+    this.subscriberFactory = subscriberFactory;
+    this.readerFactory = readerFactory;
+    this.partition = Optional.of(partition);
+  }
+
+  @Override
+  public List<? extends UnboundedSource<SequencedMessage, CheckpointMarkImpl>> split(
+      int desiredNumSplits, PipelineOptions options) throws Exception {
+    checkState(!partition.isPresent());
+    int numPartitions = PartitionLookupUtils.numPartitions(subscriberOptions.subscriptionPath());

Review Comment:
   Is there ever a crazy # of partitions? Would we want multiple partitions to a single source shard.  I believe Kafka did this.  I think one concern is per-shard information building up in the graph submitted to dataflow.



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {
+      subscriber.pop();
+    }
+    Optional<SequencedMessage> next = subscriber.peek();
+    advanced = next.isPresent();
+    if (!advanced) {
+      return false;
+    }
+    Offset nextOffset = Offset.of(next.get().getCursor().getOffset() + 1);
+    checkState(nextOffset.value() > fetchOffset.value());
+    fetchOffset = nextOffset;
+    lastMessageTimestamp = Optional.of(getTimestamp(next.get()));
+    return true;
+  }
+
+  @Override
+  public Instant getWatermark() {
+    return lastMessageTimestamp.orElse(BoundedWindow.TIMESTAMP_MIN_VALUE);
+  }
+
+  @Override
+  public CheckpointMarkImpl getCheckpointMark() {
+    subscriber.rebuffer();

Review Comment:
   add comment on why you rebuffer here



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/BlockingCommitterImpl.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static com.google.cloud.pubsublite.internal.ExtractStatus.toCanonical;
+
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.internal.wire.Committer;
+import java.util.concurrent.TimeUnit;
+
+public class BlockingCommitterImpl implements BlockingCommitter {
+
+  private final Committer committer;
+
+  BlockingCommitterImpl(Committer committer) {
+    if (!committer.isRunning()) {
+      throw new RuntimeException(committer.failureCause());

Review Comment:
   would it be easier to debug if the exception was wrapped?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r941392932


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {
+      subscriber.pop();
+    }
+    Optional<SequencedMessage> next = subscriber.peek();
+    advanced = next.isPresent();
+    if (!advanced) {
+      return false;
+    }
+    Offset nextOffset = Offset.of(next.get().getCursor().getOffset() + 1);
+    checkState(nextOffset.value() > fetchOffset.value());
+    fetchOffset = nextOffset;
+    lastMessageTimestamp = Optional.of(getTimestamp(next.get()));
+    return true;
+  }
+
+  @Override
+  public Instant getWatermark() {
+    return lastMessageTimestamp.orElse(BoundedWindow.TIMESTAMP_MIN_VALUE);
+  }
+
+  @Override
+  public CheckpointMarkImpl getCheckpointMark() {
+    subscriber.rebuffer();

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r941384587


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/UnboundedReaderImpl.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static org.apache.beam.sdk.io.gcp.pubsublite.internal.ApiServices.asCloseable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.core.ApiService.State;
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.proto.SequencedMessage;
+import com.google.protobuf.util.Timestamps;
+import java.io.IOException;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.joda.time.Instant;
+
+public class UnboundedReaderImpl extends UnboundedReader<SequencedMessage> {
+
+  private final UnboundedSource<SequencedMessage, CheckpointMarkImpl> source;
+  private final MemoryBufferedSubscriber subscriber;
+  private final TopicBacklogReader backlogReader;
+  private final BlockingCommitter committer;
+
+  private Offset fetchOffset;
+  private Optional<Instant> lastMessageTimestamp = Optional.empty();
+  private boolean advanced = false;
+
+  UnboundedReaderImpl(
+      UnboundedSource<SequencedMessage, CheckpointMarkImpl> source,
+      MemoryBufferedSubscriber subscriber,
+      TopicBacklogReader backlogReader,
+      BlockingCommitter committer,
+      Offset initialOffset) {
+    checkArgument(initialOffset.equals(subscriber.fetchOffset()));
+    this.source = source;
+    this.subscriber = subscriber;
+    this.backlogReader = backlogReader;
+    this.committer = committer;
+    this.fetchOffset = initialOffset;
+  }
+
+  @Override
+  public SequencedMessage getCurrent() throws NoSuchElementException {
+    if (!advanced) {
+      throw new NoSuchElementException();
+    }
+    return subscriber.peek().get();
+  }
+
+  @Override
+  public Instant getCurrentTimestamp() throws NoSuchElementException {
+    return getTimestamp(getCurrent());
+  }
+
+  private static Instant getTimestamp(SequencedMessage message) {
+    return Instant.ofEpochMilli(Timestamps.toMillis(message.getPublishTime()));
+  }
+
+  @Override
+  public void close() throws IOException {
+    try (AutoCloseable c1 = backlogReader;
+        AutoCloseable c2 = committer;
+        AutoCloseable c3 = asCloseable(subscriber)) {
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+  }
+
+  @Override
+  public boolean start() throws IOException {
+    try {
+      subscriber.startAsync().awaitRunning(1, TimeUnit.MINUTES);
+    } catch (Exception e) {
+      throw new IOException(e);
+    }
+    return advance();
+  }
+
+  @Override
+  public boolean advance() throws IOException {
+    if (!subscriber.state().equals(State.RUNNING)) {
+      throw new IOException("Subscriber failed: ", subscriber.failureCause());
+    }
+    if (advanced) {
+      subscriber.pop();
+    }
+    Optional<SequencedMessage> next = subscriber.peek();
+    advanced = next.isPresent();
+    if (!advanced) {
+      return false;
+    }
+    Offset nextOffset = Offset.of(next.get().getCursor().getOffset() + 1);
+    checkState(nextOffset.value() > fetchOffset.value());
+    fetchOffset = nextOffset;
+    lastMessageTimestamp = Optional.of(getTimestamp(next.get()));
+    return true;
+  }
+
+  @Override
+  public Instant getWatermark() {
+    return lastMessageTimestamp.orElse(BoundedWindow.TIMESTAMP_MIN_VALUE);

Review Comment:
   Timestamps are always increasing or equal, we reference truetime for this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r941383287


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/SubscribeTransform.java:
##########
@@ -134,19 +160,28 @@ private TopicPath getTopicPath() {
     }
   }
 
-  @Override
-  public PCollection<SequencedMessage> expand(PBegin input) {
-    PCollection<SubscriptionPartition> subscriptionPartitions;
-    subscriptionPartitions =
+  @SuppressWarnings("unused")
+  private PCollection<SequencedMessage> expandSdf(PBegin input) {
+    PCollection<SubscriptionPartition> subscriptionPartitions =
         input.apply(new SubscriptionPartitionLoader(getTopicPath(), options.subscriptionPath()));
-
     return subscriptionPartitions.apply(
         ParDo.of(
             new PerSubscriptionPartitionSdf(
-                new ManagedBacklogReaderFactoryImpl(this::newBacklogReader),
+                new ManagedFactoryImpl<>(this::newBacklogReader),
+                new ManagedFactoryImpl<>(this::newCommitter),
                 this::newInitialOffsetReader,
                 this::newRestrictionTracker,
-                this::newPartitionProcessor,
-                this::newCommitter)));
+                this::newPartitionProcessor)));
+  }
+
+  private PCollection<SequencedMessage> expandSource(PBegin input) {
+    return input.apply(
+        Read.from(
+            new UnboundedSourceImpl(options, this::newBufferedSubscriber, this::newBacklogReader)));
+  }
+
+  @Override
+  public PCollection<SequencedMessage> expand(PBegin input) {
+    return expandSource(input);

Review Comment:
   This will be deferred to a child CL which will override on runnerv2
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r941382730


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/SubscribeTransform.java:
##########
@@ -101,6 +91,42 @@ private MemoryBufferedSubscriber newBufferedSubscriber(
         consumer -> newSubscriber(subscriptionPartition.partition(), startOffset, consumer));
   }
 
+  private MemoryBufferedSubscriber getCachedSubscriber(
+      SubscriptionPartition subscriptionPartition, Offset startOffset) {
+    Supplier<MemoryBufferedSubscriber> getOrCreate =
+        () ->
+            PerServerSubscriberCache.CACHE.get(
+                subscriptionPartition,
+                () -> newBufferedSubscriber(subscriptionPartition, startOffset));
+    while (true) {
+      MemoryBufferedSubscriber subscriber = getOrCreate.get();
+      Offset fetchOffset = subscriber.fetchOffset();
+      if (startOffset.equals(fetchOffset)) {
+        return subscriber;
+      }
+      LOG.info(
+          "Discarding subscriber due to mismatch, this should be rare. {}, start: {} fetch: {}",
+          subscriptionPartition,
+          startOffset,
+          fetchOffset);
+      try {
+        subscriber.stopAsync().awaitTerminated();

Review Comment:
   The cache uses service shutdown to force evict from the cache. So yeah this will get a new one https://github.com/apache/beam/blob/0943d85113e77505f2a1291d7b256b9a4b1e5dec/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ServiceCache.java#L60



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #22612:
URL: https://github.com/apache/beam/pull/22612#issuecomment-1209453101

   Run PostCommit_Java_Dataflow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] dpcollins-google commented on a diff in pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on code in PR #22612:
URL: https://github.com/apache/beam/pull/22612#discussion_r941379594


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ApiServices.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import com.google.api.core.ApiService;
+import java.util.concurrent.TimeUnit;
+
+class ApiServices {
+
+  private ApiServices() {}
+
+  static AutoCloseable asCloseable(ApiService service) {
+    return () -> service.stopAsync().awaitTerminated(1, TimeUnit.MINUTES);

Review Comment:
   An exception is thrown if this times out- 1 minute is far, far longer than it should be waiting for, this is made to work around potential GRPC or client library bugs that cause this to block forever. 1 minute doesnt really need to be adjustable.



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/BlockingCommitterImpl.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsublite.internal;
+
+import static com.google.cloud.pubsublite.internal.ExtractStatus.toCanonical;
+
+import com.google.cloud.pubsublite.Offset;
+import com.google.cloud.pubsublite.internal.wire.Committer;
+import java.util.concurrent.TimeUnit;
+
+public class BlockingCommitterImpl implements BlockingCommitter {
+
+  private final Committer committer;
+
+  BlockingCommitterImpl(Committer committer) {
+    if (!committer.isRunning()) {
+      throw new RuntimeException(committer.failureCause());
+    }
+    this.committer = committer;
+  }
+
+  @Override
+  public void commitOffset(Offset offset) {
+    try {
+      committer.commitOffset(offset).get(1, TimeUnit.MINUTES);

Review Comment:
   See above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj merged pull request #22612: Reimplement Pub/Sub Lite's I/O using UnboundedSource.

Posted by GitBox <gi...@apache.org>.
chamikaramj merged PR #22612:
URL: https://github.com/apache/beam/pull/22612


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org