You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by al...@apache.org on 2018/02/23 11:00:21 UTC

[beam] branch master updated (51b793a -> 0259636)

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


    from 51b793a  Merge pull request #4723: [BEAM-3317] Use fixed system time for KinesisReaderTest
     new 50c6d93  Make parameter of DoFnRunners.lateDataDroppingRunner() more specific
     new c40868f  Allow overriding DoFnRunners in subclasses of Flink DoFnOperator
     new 4029e23f Invoke finishBundle() before teardown() in DoFnOperator
     new 2e0b192  [BEAM-2140] Ignore event-time timers in SplittableDoFnOperator
     new 3eecb38  [BEAM-2140] Block DoFnOperator.close() if we have pending timers
     new d15979f  [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in FlinkRunner
     new 4ec4644  Make ProcessFnRunner constructor public
     new 74ac703  [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing SDF
     new c920a39  [BEAM-2140] Enable SDF tests for Flink Streaming Runner
     new 6ba3fc1  [BEAM-2140] Enable SDF tests in gradle for Flink Streaming Runner
     new 6148a6d  [BEAM-3727] Never shutdown sources in Flink Streaming execution mode
     new 0259636  Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn ValidatesRunner tests in Flink Streaming Runner

The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/core/DoFnRunners.java  |   4 +-
 .../apache/beam/runners/core/ProcessFnRunner.java  |   2 +-
 runners/flink/build.gradle                         |  24 ++--
 runners/flink/pom.xml                              |   1 -
 .../beam/runners/flink/FlinkPipelineOptions.java   |  12 ++
 .../flink/FlinkStreamingTransformTranslators.java  |  35 +++++-
 .../apache/beam/runners/flink/TestFlinkRunner.java |   1 +
 .../wrappers/streaming/DoFnOperator.java           | 127 +++++++++++++--------
 .../wrappers/streaming/SplittableDoFnOperator.java |  16 +++
 .../wrappers/streaming/WindowDoFnOperator.java     |  18 +++
 .../streaming/io/BoundedSourceWrapper.java         |  28 +++++
 .../streaming/io/UnboundedSourceWrapper.java       |  56 ++++-----
 12 files changed, 234 insertions(+), 90 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 12/12: Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn ValidatesRunner tests in Flink Streaming Runner

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0259636661998b8aabd3bf20eff8d519bcb9fb47
Merge: 51b793a 6148a6d
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Feb 23 11:59:36 2018 +0100

    Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn ValidatesRunner tests in Flink Streaming Runner

 .../org/apache/beam/runners/core/DoFnRunners.java  |   4 +-
 .../apache/beam/runners/core/ProcessFnRunner.java  |   2 +-
 runners/flink/build.gradle                         |  24 ++--
 runners/flink/pom.xml                              |   1 -
 .../beam/runners/flink/FlinkPipelineOptions.java   |  12 ++
 .../flink/FlinkStreamingTransformTranslators.java  |  35 +++++-
 .../apache/beam/runners/flink/TestFlinkRunner.java |   1 +
 .../wrappers/streaming/DoFnOperator.java           | 127 +++++++++++++--------
 .../wrappers/streaming/SplittableDoFnOperator.java |  16 +++
 .../wrappers/streaming/WindowDoFnOperator.java     |  18 +++
 .../streaming/io/BoundedSourceWrapper.java         |  28 +++++
 .../streaming/io/UnboundedSourceWrapper.java       |  56 ++++-----
 12 files changed, 234 insertions(+), 90 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 11/12: [BEAM-3727] Never shutdown sources in Flink Streaming execution mode

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6148a6d063a0503ee435ab5084fcba3fb864b26f
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Feb 22 14:26:16 2018 +0100

    [BEAM-3727] Never shutdown sources in Flink Streaming execution mode
    
    This adds an option that controls whether to shutdown sources or not in
    case of reaching the +Inf watermark.
    
    The reason for this is https://issues.apache.org/jira/browse/FLINK-2491,
    which causes checkpointing to stop once some source is shut down.
---
 .../beam/runners/flink/FlinkPipelineOptions.java   | 12 +++++
 .../apache/beam/runners/flink/TestFlinkRunner.java |  1 +
 .../streaming/io/BoundedSourceWrapper.java         | 28 +++++++++++
 .../streaming/io/UnboundedSourceWrapper.java       | 56 ++++++++++++----------
 4 files changed, 71 insertions(+), 26 deletions(-)

diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
index 01f7847..b2cbefb 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
@@ -137,4 +137,16 @@ public interface FlinkPipelineOptions
   Long getMaxBundleTimeMills();
   void setMaxBundleTimeMills(Long time);
 
+  /**
+   * Whether to shutdown sources when their watermark reaches {@code +Inf}. For production use
+   * cases you want this to be disabled because Flink will currently (versions {@literal <=} 1.5)
+   * stop doing checkpoints when any operator (which includes sources) is finished.
+   *
+   * <p>Please see <a href="https://issues.apache.org/jira/browse/FLINK-2491">FLINK-2491</a> for
+   * progress on this issue.
+   */
+  @Description("If set, shutdown sources when their watermark reaches +Inf.")
+  @Default.Boolean(false)
+  Boolean isShutdownSourcesOnFinalWatermark();
+  void setShutdownSourcesOnFinalWatermark(Boolean shutdownOnFinalWatermark);
 }
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
index 01b67e5..47d4494 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
@@ -36,6 +36,7 @@ public class TestFlinkRunner extends PipelineRunner<PipelineResult> {
   private TestFlinkRunner(FlinkPipelineOptions options) {
     // We use [auto] for testing since this will make it pick up the Testing ExecutionEnvironment
     options.setFlinkMaster("[auto]");
+    options.setShutdownSourcesOnFinalWatermark(true);
     this.delegate = FlinkRunner.fromOptions(options);
   }
 
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
index 5ddc46f..6db5426 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
@@ -21,6 +21,7 @@ import com.google.common.annotations.VisibleForTesting;
 import java.util.ArrayList;
 import java.util.List;
 import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.flink.FlinkPipelineOptions;
 import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
 import org.apache.beam.runners.flink.metrics.ReaderInvocationUtil;
 import org.apache.beam.sdk.io.BoundedSource;
@@ -180,6 +181,33 @@ public class BoundedSourceWrapper<OutputT>
 
     // emit final Long.MAX_VALUE watermark, just to be sure
     ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
+
+    FlinkPipelineOptions options = serializedOptions.get().as(FlinkPipelineOptions.class);
+    if (!options.isShutdownSourcesOnFinalWatermark()) {
+      // do nothing, but still look busy ...
+      // we can't return here since Flink requires that all operators stay up,
+      // otherwise checkpointing would not work correctly anymore
+      //
+      // See https://issues.apache.org/jira/browse/FLINK-2491 for progress on this issue
+
+      // wait until this is canceled
+      final Object waitLock = new Object();
+      while (isRunning) {
+        try {
+          // Flink will interrupt us at some point
+          //noinspection SynchronizationOnLocalVariableOrMethodParameter
+          synchronized (waitLock) {
+            // don't wait indefinitely, in case something goes horribly wrong
+            waitLock.wait(1000);
+          }
+        } catch (InterruptedException e) {
+          if (!isRunning) {
+            // restore the interrupted state, and fall through the loop
+            Thread.currentThread().interrupt();
+          }
+        }
+      }
+    }
   }
 
   /**
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
index 817dd74..fc23c01 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
@@ -23,6 +23,7 @@ import java.util.Iterator;
 import java.util.LinkedHashMap;
 import java.util.List;
 import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.flink.FlinkPipelineOptions;
 import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
 import org.apache.beam.runners.flink.metrics.ReaderInvocationUtil;
 import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
@@ -224,32 +225,7 @@ public class UnboundedSourceWrapper<
             serializedOptions.get(),
             metricContainer);
 
-    if (localReaders.size() == 0) {
-      // do nothing, but still look busy ...
-      // also, output a Long.MAX_VALUE watermark since we know that we're not
-      // going to emit anything
-      // we can't return here since Flink requires that all operators stay up,
-      // otherwise checkpointing would not work correctly anymore
-      ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
-
-      // wait until this is canceled
-      final Object waitLock = new Object();
-      while (isRunning) {
-        try {
-          // Flink will interrupt us at some point
-          //noinspection SynchronizationOnLocalVariableOrMethodParameter
-          synchronized (waitLock) {
-            // don't wait indefinitely, in case something goes horribly wrong
-            waitLock.wait(1000);
-          }
-        } catch (InterruptedException e) {
-          if (!isRunning) {
-            // restore the interrupted state, and fall through the loop
-            Thread.currentThread().interrupt();
-          }
-        }
-      }
-    } else if (localReaders.size() == 1) {
+    if (localReaders.size() == 1) {
       // the easy case, we just read from one reader
       UnboundedSource.UnboundedReader<OutputT> reader = localReaders.get(0);
 
@@ -305,7 +281,35 @@ public class UnboundedSourceWrapper<
           hadData = false;
         }
       }
+    }
 
+    ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
+
+    FlinkPipelineOptions options = serializedOptions.get().as(FlinkPipelineOptions.class);
+    if (!options.isShutdownSourcesOnFinalWatermark()) {
+      // do nothing, but still look busy ...
+      // we can't return here since Flink requires that all operators stay up,
+      // otherwise checkpointing would not work correctly anymore
+      //
+      // See https://issues.apache.org/jira/browse/FLINK-2491 for progress on this issue
+
+      // wait until this is canceled
+      final Object waitLock = new Object();
+      while (isRunning) {
+        try {
+          // Flink will interrupt us at some point
+          //noinspection SynchronizationOnLocalVariableOrMethodParameter
+          synchronized (waitLock) {
+            // don't wait indefinitely, in case something goes horribly wrong
+            waitLock.wait(1000);
+          }
+        } catch (InterruptedException e) {
+          if (!isRunning) {
+            // restore the interrupted state, and fall through the loop
+            Thread.currentThread().interrupt();
+          }
+        }
+      }
     }
   }
 

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 07/12: Make ProcessFnRunner constructor public

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4ec4644952c05167f52ae0937e043140d26059ae
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Feb 8 13:01:22 2018 +0100

    Make ProcessFnRunner constructor public
    
    We need this to be able to instantiate with the constructor in the Flink
    Runner.
---
 .../src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java     | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
index 88275d6..e4dfd13 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
@@ -43,7 +43,7 @@ public class ProcessFnRunner<InputT, OutputT, RestrictionT>
   private final Collection<PCollectionView<?>> views;
   private final ReadyCheckingSideInputReader sideInputReader;
 
-  ProcessFnRunner(
+  public ProcessFnRunner(
       DoFnRunner<KeyedWorkItem<String, KV<InputT, RestrictionT>>, OutputT> underlying,
       Collection<PCollectionView<?>> views,
       ReadyCheckingSideInputReader sideInputReader) {

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 06/12: [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in FlinkRunner

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d15979f2cf0d6bc47d049a6ea157d9d7b2b97848
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Wed Jan 10 06:48:49 2018 +0100

    [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in FlinkRunner
---
 .../translation/wrappers/streaming/SplittableDoFnOperator.java | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
index 44be5f3..e088b07 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
@@ -26,6 +26,7 @@ import java.util.Map;
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
+import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.KeyedWorkItem;
 import org.apache.beam.runners.core.KeyedWorkItems;
 import org.apache.beam.runners.core.OutputAndTimeBoundedSplittableProcessElementInvoker;
@@ -88,6 +89,15 @@ public class SplittableDoFnOperator<
   }
 
   @Override
+  protected DoFnRunner<
+      KeyedWorkItem<String, KV<InputT, RestrictionT>>, OutputT> createWrappingDoFnRunner(
+          DoFnRunner<KeyedWorkItem<String, KV<InputT, RestrictionT>>, OutputT> wrappedRunner) {
+    // don't wrap in anything because we don't need state cleanup because ProcessFn does
+    // all that
+    return wrappedRunner;
+  }
+
+  @Override
   public void open() throws Exception {
     super.open();
 

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 08/12: [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing SDF

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 74ac703bbfa40d16e7d1115912768c7a63598d52
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Feb 8 13:02:26 2018 +0100

    [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing SDF
    
    For this to work, we need to also change how we wrap values in
    KeyedWorkItems, because ProcessFnRunner expects them to be in the
    GlobalWindow.
---
 .../flink/FlinkStreamingTransformTranslators.java  | 35 ++++++++++++++++++----
 .../wrappers/streaming/DoFnOperator.java           | 11 +++++--
 2 files changed, 39 insertions(+), 7 deletions(-)

diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index d39b5c1..f5dc3ce 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -1017,10 +1017,8 @@ class FlinkStreamingTransformTranslators {
 
 
       WindowedValue.
-          FullWindowedValueCoder<SingletonKeyedWorkItem<K, InputT>> windowedWorkItemCoder =
-          WindowedValue.getFullCoder(
-              workItemCoder,
-              input.getWindowingStrategy().getWindowFn().windowCoder());
+          ValueOnlyWindowedValueCoder<SingletonKeyedWorkItem<K, InputT>> windowedWorkItemCoder =
+          WindowedValue.getValueOnlyCoder(workItemCoder);
 
       CoderTypeInformation<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemTypeInfo =
           new CoderTypeInformation<>(windowedWorkItemCoder);
@@ -1029,7 +1027,7 @@ class FlinkStreamingTransformTranslators {
 
       DataStream<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemStream =
           inputDataStream
-              .flatMap(new ToKeyedWorkItem<>())
+              .flatMap(new ToKeyedWorkItemInGlobalWindow<>())
               .returns(workItemTypeInfo)
               .name("ToKeyedWorkItem");
 
@@ -1041,6 +1039,33 @@ class FlinkStreamingTransformTranslators {
     }
   }
 
+  private static class ToKeyedWorkItemInGlobalWindow<K, InputT>
+      extends RichFlatMapFunction<
+      WindowedValue<KV<K, InputT>>,
+      WindowedValue<SingletonKeyedWorkItem<K, InputT>>> {
+
+    @Override
+    public void flatMap(
+        WindowedValue<KV<K, InputT>> inWithMultipleWindows,
+        Collector<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> out) throws Exception {
+
+      // we need to wrap each one work item per window for now
+      // since otherwise the PushbackSideInputRunner will not correctly
+      // determine whether side inputs are ready
+      //
+      // this is tracked as https://issues.apache.org/jira/browse/BEAM-1850
+      for (WindowedValue<KV<K, InputT>> in : inWithMultipleWindows.explodeWindows()) {
+        SingletonKeyedWorkItem<K, InputT> workItem =
+            new SingletonKeyedWorkItem<>(
+                in.getValue().getKey(),
+                in.withValue(in.getValue().getValue()));
+
+        out.collect(WindowedValue.valueInGlobalWindow(workItem));
+      }
+    }
+  }
+
+
   private static class FlattenPCollectionTranslator<T>
       extends FlinkStreamingPipelineTranslator.StreamTransformTranslator<
         PTransform<PCollection<T>, PCollection<T>>> {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 37f56f5..f9b4ee3 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -39,10 +39,12 @@ import javax.annotation.Nullable;
 import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.DoFnRunners;
 import org.apache.beam.runners.core.NullSideInputReader;
+import org.apache.beam.runners.core.ProcessFnRunner;
 import org.apache.beam.runners.core.PushbackSideInputDoFnRunner;
 import org.apache.beam.runners.core.SideInputHandler;
 import org.apache.beam.runners.core.SideInputReader;
 import org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner;
+import org.apache.beam.runners.core.SplittableParDoViaKeyedWorkItems;
 import org.apache.beam.runners.core.StateInternals;
 import org.apache.beam.runners.core.StateNamespace;
 import org.apache.beam.runners.core.StateNamespaces;
@@ -352,8 +354,13 @@ public class DoFnOperator<InputT, OutputT>
             .scheduleAtFixedRate(
                 timestamp -> checkInvokeFinishBundleByTime(), bundleCheckPeriod, bundleCheckPeriod);
 
-    pushbackDoFnRunner =
-        SimplePushbackSideInputDoFnRunner.create(doFnRunner, sideInputs, sideInputHandler);
+    if (doFn instanceof SplittableParDoViaKeyedWorkItems.ProcessFn) {
+      pushbackDoFnRunner =
+          new ProcessFnRunner<>((DoFnRunner) doFnRunner, sideInputs, sideInputHandler);
+    } else {
+      pushbackDoFnRunner =
+          SimplePushbackSideInputDoFnRunner.create(doFnRunner, sideInputs, sideInputHandler);
+    }
   }
 
   @Override

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 09/12: [BEAM-2140] Enable SDF tests for Flink Streaming Runner

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c920a39b9781ac604f55059e597e327643d58205
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Jan 5 14:20:19 2018 +0100

    [BEAM-2140] Enable SDF tests for Flink Streaming Runner
---
 runners/flink/pom.xml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index 46917e9..cd4d8c3 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -94,7 +94,6 @@
                     org.apache.beam.sdk.testing.LargeKeys$Above100MB,
                     org.apache.beam.sdk.testing.UsesCommittedMetrics,
                     org.apache.beam.sdk.testing.UsesTestStream,
-                    org.apache.beam.sdk.testing.UsesSplittableParDo
                   </excludedGroups>
                   <parallel>none</parallel>
                   <failIfNoTests>true</failIfNoTests>

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 10/12: [BEAM-2140] Enable SDF tests in gradle for Flink Streaming Runner

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6ba3fc1ff2798764756a7188980facc31eb1ac4c
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Feb 8 13:00:34 2018 +0100

    [BEAM-2140] Enable SDF tests in gradle for Flink Streaming Runner
---
 runners/flink/build.gradle | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/runners/flink/build.gradle b/runners/flink/build.gradle
index dc77bcf..362288e 100644
--- a/runners/flink/build.gradle
+++ b/runners/flink/build.gradle
@@ -97,13 +97,23 @@ def createValidatesRunnerTask(Map m) {
     classpath = configurations.validatesRunner
     testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
     maxParallelForks 4
-    useJUnit {
-      includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
-      excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
-      excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
-      excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDo'
-      excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
-      excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+    if (config.streaming) {
+      useJUnit {
+        includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+        excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+        excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+        excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+        excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+      }
+    } else {
+      useJUnit {
+        includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+        excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+        excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+        excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDo'
+        excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+        excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+      }
     }
   }
 }

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 03/12: Invoke finishBundle() before teardown() in DoFnOperator

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4029e23f3075365eddcdad5187b4fc8a6e590989
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Jan 5 14:22:40 2018 +0100

    Invoke finishBundle() before teardown() in DoFnOperator
---
 .../runners/flink/translation/wrappers/streaming/DoFnOperator.java     | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 2e7f741..688a7cb 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -363,6 +363,9 @@ public class DoFnOperator<InputT, OutputT>
       super.dispose();
       checkFinishBundleTimer.cancel(true);
     } finally {
+      if (bundleStarted) {
+        invokeFinishBundle();
+      }
       doFnInvoker.invokeTeardown();
     }
   }

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 01/12: Make parameter of DoFnRunners.lateDataDroppingRunner() more specific

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 50c6d9316c8c5089065dd291b097e06b0c80980e
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Jan 11 10:56:49 2018 +0100

    Make parameter of DoFnRunners.lateDataDroppingRunner() more specific
---
 .../src/main/java/org/apache/beam/runners/core/DoFnRunners.java       | 4 ++--
 .../runners/flink/translation/wrappers/streaming/DoFnOperator.java    | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
index 9d3e25d..80c830a 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
@@ -80,12 +80,12 @@ public class DoFnRunners {
   public static <K, InputT, OutputT, W extends BoundedWindow>
       DoFnRunner<KeyedWorkItem<K, InputT>, KV<K, OutputT>> lateDataDroppingRunner(
           DoFnRunner<KeyedWorkItem<K, InputT>, KV<K, OutputT>> wrappedRunner,
-          StepContext stepContext,
+          TimerInternals timerInternals,
           WindowingStrategy<?, W> windowingStrategy) {
     return new LateDataDroppingDoFnRunner<>(
         wrappedRunner,
         windowingStrategy,
-        stepContext.timerInternals());
+        timerInternals);
   }
 
   /**
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 41a35ce..8ccbd8f 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -314,7 +314,7 @@ public class DoFnOperator<InputT, OutputT>
 
       doFnRunner = DoFnRunners.lateDataDroppingRunner(
           (DoFnRunner) doFnRunner,
-          stepContext,
+          timerInternals,
           windowingStrategy);
     } else if (keyCoder != null) {
       // It is a stateful DoFn

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 04/12: [BEAM-2140] Ignore event-time timers in SplittableDoFnOperator

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2e0b192efbae72fe5c0ef61abfc7b4d1b3bb75f5
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Jan 5 14:16:27 2018 +0100

    [BEAM-2140] Ignore event-time timers in SplittableDoFnOperator
    
    SplittableDoFnOperator is only used for executing ProcessFn, which does
    not use event-time timers. However, StatefulDoFnRunner does use
    event-time timers for state cleanup so this change makes sure that they
    don't end up being forwarded to the ProcessFn.
---
 .../runners/flink/translation/wrappers/streaming/DoFnOperator.java  | 1 -
 .../translation/wrappers/streaming/SplittableDoFnOperator.java      | 6 ++++++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 688a7cb..dd2f9c4 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -38,7 +38,6 @@ import java.util.concurrent.ScheduledFuture;
 import javax.annotation.Nullable;
 import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.DoFnRunners;
-import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetNewDoFn;
 import org.apache.beam.runners.core.NullSideInputReader;
 import org.apache.beam.runners.core.PushbackSideInputDoFnRunner;
 import org.apache.beam.runners.core.SideInputHandler;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
index 1a418a0..44be5f3 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
@@ -37,6 +37,7 @@ import org.apache.beam.runners.core.TimerInternals;
 import org.apache.beam.runners.core.TimerInternalsFactory;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.state.TimeDomain;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -142,6 +143,11 @@ public class SplittableDoFnOperator<
 
   @Override
   public void fireTimer(InternalTimer<?, TimerInternals.TimerData> timer) {
+    if (timer.getNamespace().getDomain().equals(TimeDomain.EVENT_TIME)) {
+      // ignore this, it can only be a state cleanup timers from StatefulDoFnRunner and ProcessFn
+      // does its own state cleanup and should never set event-time timers.
+      return;
+    }
     doFnRunner.processElement(
         WindowedValue.valueInGlobalWindow(
             KeyedWorkItems.timersWorkItem(

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 02/12: Allow overriding DoFnRunners in subclasses of Flink DoFnOperator

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c40868fe2fc550ae97c7b2d9308dd8b58b20edab
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Jan 11 10:52:01 2018 +0100

    Allow overriding DoFnRunners in subclasses of Flink DoFnOperator
---
 .../wrappers/streaming/DoFnOperator.java           | 69 +++++++++++-----------
 .../wrappers/streaming/WindowDoFnOperator.java     | 18 ++++++
 2 files changed, 52 insertions(+), 35 deletions(-)

diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 8ccbd8f..2e7f741 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -218,6 +218,39 @@ public class DoFnOperator<InputT, OutputT>
     return doFn;
   }
 
+  // allow overriding this, for example SplittableDoFnOperator will not create a
+  // stateful DoFn runner because ProcessFn, which is used for executing a Splittable DoFn
+  // doesn't play by the normal DoFn rules and WindowDoFnOperator uses LateDataDroppingDoFnRunner
+  protected DoFnRunner<InputT, OutputT> createWrappingDoFnRunner(
+      DoFnRunner<InputT, OutputT> wrappedRunner) {
+
+    if (keyCoder != null) {
+      StatefulDoFnRunner.CleanupTimer cleanupTimer =
+          new StatefulDoFnRunner.TimeInternalsCleanupTimer(
+              timerInternals, windowingStrategy);
+
+      // we don't know the window type
+      @SuppressWarnings({"unchecked", "rawtypes"})
+      Coder windowCoder = windowingStrategy.getWindowFn().windowCoder();
+
+      @SuppressWarnings({"unchecked", "rawtypes"})
+      StatefulDoFnRunner.StateCleaner<?> stateCleaner =
+          new StatefulDoFnRunner.StateInternalsStateCleaner<>(
+              doFn, keyedStateInternals, windowCoder);
+
+
+      return DoFnRunners.defaultStatefulDoFnRunner(
+          doFn,
+          wrappedRunner,
+          windowingStrategy,
+          cleanupTimer,
+          stateCleaner);
+
+    } else {
+      return doFnRunner;
+    }
+  }
+
   @Override
   public void setup(
       StreamTask<?, ?> containingTask,
@@ -304,41 +337,7 @@ public class DoFnOperator<InputT, OutputT>
         stepContext,
         windowingStrategy);
 
-    if (doFn instanceof GroupAlsoByWindowViaWindowSetNewDoFn) {
-      // When the doFn is this, we know it came from WindowDoFnOperator and
-      //   InputT = KeyedWorkItem<K, V>
-      //   OutputT = KV<K, V>
-      //
-      // for some K, V
-
-
-      doFnRunner = DoFnRunners.lateDataDroppingRunner(
-          (DoFnRunner) doFnRunner,
-          timerInternals,
-          windowingStrategy);
-    } else if (keyCoder != null) {
-      // It is a stateful DoFn
-
-      StatefulDoFnRunner.CleanupTimer cleanupTimer =
-          new StatefulDoFnRunner.TimeInternalsCleanupTimer(
-              stepContext.timerInternals(), windowingStrategy);
-
-      // we don't know the window type
-      @SuppressWarnings({"unchecked", "rawtypes"})
-      Coder windowCoder = windowingStrategy.getWindowFn().windowCoder();
-
-      @SuppressWarnings({"unchecked", "rawtypes"})
-      StatefulDoFnRunner.StateCleaner<?> stateCleaner =
-          new StatefulDoFnRunner.StateInternalsStateCleaner<>(
-              doFn, stepContext.stateInternals(), windowCoder);
-
-      doFnRunner = DoFnRunners.defaultStatefulDoFnRunner(
-          doFn,
-          doFnRunner,
-          windowingStrategy,
-          cleanupTimer,
-          stateCleaner);
-    }
+    doFnRunner = createWrappingDoFnRunner(doFnRunner);
 
     if (options.getEnableMetrics()) {
       doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
index 7a04238..8447ade 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
@@ -23,6 +23,8 @@ import java.util.Collection;
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
+import org.apache.beam.runners.core.DoFnRunner;
+import org.apache.beam.runners.core.DoFnRunners;
 import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetNewDoFn;
 import org.apache.beam.runners.core.KeyedWorkItem;
 import org.apache.beam.runners.core.KeyedWorkItems;
@@ -79,6 +81,22 @@ public class WindowDoFnOperator<K, InputT, OutputT>
   }
 
   @Override
+  protected DoFnRunner<KeyedWorkItem<K, InputT>, KV<K, OutputT>> createWrappingDoFnRunner(
+      DoFnRunner<KeyedWorkItem<K, InputT>, KV<K, OutputT>> wrappedRunner) {
+    // When the doFn is this, we know it came from WindowDoFnOperator and
+    //   InputT = KeyedWorkItem<K, V>
+    //   OutputT = KV<K, V>
+    //
+    // for some K, V
+
+
+    return DoFnRunners.lateDataDroppingRunner(
+        (DoFnRunner) doFnRunner,
+        timerInternals,
+        windowingStrategy);
+  }
+
+  @Override
   protected DoFn<KeyedWorkItem<K, InputT>, KV<K, OutputT>> getDoFn() {
     StateInternalsFactory<K> stateInternalsFactory =
         key -> {

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.

[beam] 05/12: [BEAM-2140] Block DoFnOperator.close() if we have pending timers

Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3eecb381f3fb412df6844cfc13b12bf265253926
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Jan 5 14:17:49 2018 +0100

    [BEAM-2140] Block DoFnOperator.close() if we have pending timers
    
    It can happen that the input operation finishes while we still have
    pending processing-time timers, for example from processing a Splittable
    DoFn. This change makes sure that we block as long as we have pending
    timers.
    
    This change also makes sure that we forward a +Inf watermark in close().
    We have to do this because it can happen that we get a +Inf watermark on
    input while we still have active watermark holds (which will get
    resolved when all pending timers are gone). With this change we make
    sure to send a +Inf watermark downstream once everything is resolved.
---
 .../wrappers/streaming/DoFnOperator.java           | 43 +++++++++++++++++-----
 1 file changed, 33 insertions(+), 10 deletions(-)

diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index dd2f9c4..37f56f5 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -371,18 +371,41 @@ public class DoFnOperator<InputT, OutputT>
 
   @Override
   public void close() throws Exception {
-    super.close();
-
-    // sanity check: these should have been flushed out by +Inf watermarks
-    if (!sideInputs.isEmpty() && nonKeyedStateInternals != null) {
-      BagState<WindowedValue<InputT>> pushedBack =
-          nonKeyedStateInternals.state(StateNamespaces.global(), pushedBackTag);
+    try {
 
-      Iterable<WindowedValue<InputT>> pushedBackContents = pushedBack.read();
-      if (pushedBackContents != null && !Iterables.isEmpty(pushedBackContents)) {
-        String pushedBackString = Joiner.on(",").join(pushedBackContents);
+      // This is our last change to block shutdown of this operator while
+      // there are still remaining processing-time timers. Flink will ignore pending
+      // processing-time timers when upstream operators have shut down and will also
+      // shut down this operator with pending processing-time timers.
+      while (this.numProcessingTimeTimers() > 0) {
+        getContainingTask().getCheckpointLock().wait(100);
+      }
+      if (this.numProcessingTimeTimers() > 0) {
         throw new RuntimeException(
-            "Leftover pushed-back data: " + pushedBackString + ". This indicates a bug.");
+            "There are still processing-time timers left, this indicates a bug");
+      }
+
+      // make sure we send a +Inf watermark downstream. It can happen that we receive +Inf
+      // in processWatermark*() but have holds, so we have to re-evaluate here.
+      processWatermark(new Watermark(Long.MAX_VALUE));
+      if (currentOutputWatermark < Long.MAX_VALUE) {
+        throw new RuntimeException("There are still watermark holds. Watermark held at "
+            + keyedStateInternals.watermarkHold().getMillis() + ".");
+      }
+    } finally {
+      super.close();
+
+      // sanity check: these should have been flushed out by +Inf watermarks
+      if (!sideInputs.isEmpty() && nonKeyedStateInternals != null) {
+        BagState<WindowedValue<InputT>> pushedBack =
+            nonKeyedStateInternals.state(StateNamespaces.global(), pushedBackTag);
+
+        Iterable<WindowedValue<InputT>> pushedBackContents = pushedBack.read();
+        if (pushedBackContents != null && !Iterables.isEmpty(pushedBackContents)) {
+          String pushedBackString = Joiner.on(",").join(pushedBackContents);
+          throw new RuntimeException(
+              "Leftover pushed-back data: " + pushedBackString + ". This indicates a bug.");
+        }
       }
     }
   }

-- 
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.