You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/11/03 13:35:00 UTC

[jira] [Work logged] (BEAM-10703) Add support for auto-sharded GroupIntoBatches in Dataflow runner

     [ https://issues.apache.org/jira/browse/BEAM-10703?focusedWorklogId=506684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-506684 ]

ASF GitHub Bot logged work on BEAM-10703:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 03/Nov/20 13:34
            Start Date: 03/Nov/20 13:34
    Worklog Time Spent: 10m 
      Work Description: robertwb commented on a change in pull request #13208:
URL: https://github.com/apache/beam/pull/13208#discussion_r516178126



##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/GroupIntoBatchesOverride.java
##########
@@ -92,9 +96,58 @@ public void process(ProcessContext c) {
     }
   }
 
+  static class BatchGroupIntoBatchesWithShardedKeyOverrideFactory<K, V>

Review comment:
       Yeah, GBK before batch stateful DoFns should definitely be done on the runner. 
   
   Should we be using the GBK + Iterators.partition on all runners, not just Dataflow?  

##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/GroupIntoBatchesOverride.java
##########
@@ -103,43 +156,76 @@ public void process(ProcessContext c) {
     }
 
     @Override
-    public PTransformReplacement<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>>
+    public PTransformReplacement<PCollection<KV<K, V>>, PCollection<KV<ShardedKey<K>, Iterable<V>>>>
         getReplacementTransform(
             AppliedPTransform<
-                    PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>, GroupIntoBatches<K, V>>
+                    PCollection<KV<K, V>>,
+                    PCollection<KV<ShardedKey<K>, Iterable<V>>>,
+                    GroupIntoBatches<K, V>.WithShardedKey>
                 transform) {
       return PTransformReplacement.of(
           PTransformReplacements.getSingletonMainInput(transform),
-          new StreamingGroupIntoBatches(runner, transform.getTransform()));
+          new StreamingGroupIntoBatchesWithShardedKey<>(runner, transform.getTransform()));
     }
 
     @Override
     public Map<PCollection<?>, ReplacementOutput> mapOutputs(
-        Map<TupleTag<?>, PCollection<?>> outputs, PCollection<KV<K, Iterable<V>>> newOutput) {
+        Map<TupleTag<?>, PCollection<?>> outputs,
+        PCollection<KV<ShardedKey<K>, Iterable<V>>> newOutput) {
       return ReplacementOutputs.singleton(outputs, newOutput);
     }
   }
 
   /**
-   * Specialized implementation of {@link GroupIntoBatches} for unbounded Dataflow pipelines. The
-   * override does the same thing as the original transform but additionally record the input to add
-   * corresponding properties during the graph translation.
+   * Specialized implementation of {@link GroupIntoBatches.WithShardedKey} for unbounded Dataflow
+   * pipelines. The override does the same thing as the original transform but additionally records
+   * the input of {@code GroupIntoBatchesDoFn} in order to append relevant step properties during
+   * the graph translation.
    */
-  static class StreamingGroupIntoBatches<K, V>
-      extends PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> {
+  static class StreamingGroupIntoBatchesWithShardedKey<K, V>
+      extends PTransform<PCollection<KV<K, V>>, PCollection<KV<ShardedKey<K>, Iterable<V>>>> {
 
     private final transient DataflowRunner runner;
-    private final GroupIntoBatches<K, V> original;
+    private final GroupIntoBatches<K, V>.WithShardedKey original;
 
-    public StreamingGroupIntoBatches(DataflowRunner runner, GroupIntoBatches<K, V> original) {
+    public StreamingGroupIntoBatchesWithShardedKey(
+        DataflowRunner runner, GroupIntoBatches<K, V>.WithShardedKey original) {
       this.runner = runner;
       this.original = original;
     }
 
     @Override
-    public PCollection<KV<K, Iterable<V>>> expand(PCollection<KV<K, V>> input) {
-      runner.maybeRecordPCollectionWithAutoSharding(input);
-      return input.apply(original);
+    public PCollection<KV<ShardedKey<K>, Iterable<V>>> expand(PCollection<KV<K, V>> input) {
+      PCollection<KV<ShardedKey<K>, V>> intermediate_input = ShardKeys(input);
+
+      runner.maybeRecordPCollectionWithAutoSharding(intermediate_input);
+
+      if (original.getMaxBufferingDuration() != null) {

Review comment:
       Sounds like privacy is getting in the way of writing clean code here. What about making it package private, and adding a (package) helper to identify it? 

##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/GroupIntoBatchesOverride.java
##########
@@ -210,20 +211,23 @@ public StreamingGroupIntoBatchesWithShardedKey(
     }
   }
 
-  private static <K, V> PCollection<KV<ShardedKey<K>, V>> ShardKeys(PCollection<KV<K, V>> input) {
+  private static <K, V> PCollection<KV<ShardedKey<K>, V>> shardKeys(PCollection<KV<K, V>> input) {
     KvCoder<K, V> inputCoder = (KvCoder<K, V>) input.getCoder();
     org.apache.beam.sdk.coders.Coder<K> keyCoder =
         (org.apache.beam.sdk.coders.Coder<K>) inputCoder.getCoderArguments().get(0);
     org.apache.beam.sdk.coders.Coder<V> valueCoder =
         (org.apache.beam.sdk.coders.Coder<V>) inputCoder.getCoderArguments().get(1);
     return input
         .apply(
-            "ShardKeys",
+            "Shard Keys",
             MapElements.via(
                 new SimpleFunction<KV<K, V>, KV<ShardedKey<K>, V>>() {
                   @Override
                   public KV<ShardedKey<K>, V> apply(KV<K, V> input) {
-                    return KV.of(ShardedKey.of(input.getKey(), new byte[0]), input.getValue());
+                    long tid = Thread.currentThread().getId();

Review comment:
       Thanks. These are likely to be re-used for all workers. Can you add in a statically initialized (for each worker) random long as well? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 506684)
    Time Spent: 9h  (was: 8h 50m)

> Add support for auto-sharded GroupIntoBatches in Dataflow runner
> ----------------------------------------------------------------
>
>                 Key: BEAM-10703
>                 URL: https://issues.apache.org/jira/browse/BEAM-10703
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-dataflow
>            Reporter: Siyuan Chen
>            Assignee: Siyuan Chen
>            Priority: P1
>          Time Spent: 9h
>  Remaining Estimate: 0h
>
> The proposal of improving GroupIntoBatches transform is in BEAM-10475
> This tracks the support in Cloud Dataflow Runner.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)