You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/10/21 20:21:48 UTC
[GitHub] [beam] boyuanzz commented on a change in pull request #13161: fix: Fix split() logic for PubsubLiteIO
boyuanzz commented on a change in pull request #13161:
URL: https://github.com/apache/beam/pull/13161#discussion_r509652035
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteUnboundedSource.java
##########
@@ -53,9 +53,17 @@
@Override
public List<? extends UnboundedSource<SequencedMessage, OffsetCheckpointMark>> split(
int desiredNumSplits, PipelineOptions options) {
+ ArrayList<ArrayList<Partition>> partitionPartitions = new ArrayList<>(desiredNumSplits);
Review comment:
We can use `min(desiredNumSplits, subscriberOptions.partitions().length())` as the actual nums of split.
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteUnboundedSource.java
##########
@@ -53,9 +53,17 @@
@Override
public List<? extends UnboundedSource<SequencedMessage, OffsetCheckpointMark>> split(
int desiredNumSplits, PipelineOptions options) {
+ ArrayList<ArrayList<Partition>> partitionPartitions = new ArrayList<>(desiredNumSplits);
+ for (int i = 0; i < desiredNumSplits; i++) {
+ partitionPartitions.add(new ArrayList<>());
+ }
+ int counter = 0;
+ for (Partition partition : subscriberOptions.partitions()) {
Review comment:
We can use `for (int index = 0 ; index < length; index++)` to get ride of `counter`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org