You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by al...@apache.org on 2018/02/23 11:00:21 UTC
[beam] branch master updated (51b793a -> 0259636)
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 51b793a Merge pull request #4723: [BEAM-3317] Use fixed system time for KinesisReaderTest
new 50c6d93 Make parameter of DoFnRunners.lateDataDroppingRunner() more specific
new c40868f Allow overriding DoFnRunners in subclasses of Flink DoFnOperator
new 4029e23f Invoke finishBundle() before teardown() in DoFnOperator
new 2e0b192 [BEAM-2140] Ignore event-time timers in SplittableDoFnOperator
new 3eecb38 [BEAM-2140] Block DoFnOperator.close() if we have pending timers
new d15979f [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in FlinkRunner
new 4ec4644 Make ProcessFnRunner constructor public
new 74ac703 [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing SDF
new c920a39 [BEAM-2140] Enable SDF tests for Flink Streaming Runner
new 6ba3fc1 [BEAM-2140] Enable SDF tests in gradle for Flink Streaming Runner
new 6148a6d [BEAM-3727] Never shutdown sources in Flink Streaming execution mode
new 0259636 Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn ValidatesRunner tests in Flink Streaming Runner
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/beam/runners/core/DoFnRunners.java | 4 +-
.../apache/beam/runners/core/ProcessFnRunner.java | 2 +-
runners/flink/build.gradle | 24 ++--
runners/flink/pom.xml | 1 -
.../beam/runners/flink/FlinkPipelineOptions.java | 12 ++
.../flink/FlinkStreamingTransformTranslators.java | 35 +++++-
.../apache/beam/runners/flink/TestFlinkRunner.java | 1 +
.../wrappers/streaming/DoFnOperator.java | 127 +++++++++++++--------
.../wrappers/streaming/SplittableDoFnOperator.java | 16 +++
.../wrappers/streaming/WindowDoFnOperator.java | 18 +++
.../streaming/io/BoundedSourceWrapper.java | 28 +++++
.../streaming/io/UnboundedSourceWrapper.java | 56 ++++-----
12 files changed, 234 insertions(+), 90 deletions(-)
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 12/12: Merge pull request #4639: [BEAM-2140] Fix
SplittableDoFn ValidatesRunner tests in Flink Streaming Runner
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 0259636661998b8aabd3bf20eff8d519bcb9fb47
Merge: 51b793a 6148a6d
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Feb 23 11:59:36 2018 +0100
Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn ValidatesRunner tests in Flink Streaming Runner
.../org/apache/beam/runners/core/DoFnRunners.java | 4 +-
.../apache/beam/runners/core/ProcessFnRunner.java | 2 +-
runners/flink/build.gradle | 24 ++--
runners/flink/pom.xml | 1 -
.../beam/runners/flink/FlinkPipelineOptions.java | 12 ++
.../flink/FlinkStreamingTransformTranslators.java | 35 +++++-
.../apache/beam/runners/flink/TestFlinkRunner.java | 1 +
.../wrappers/streaming/DoFnOperator.java | 127 +++++++++++++--------
.../wrappers/streaming/SplittableDoFnOperator.java | 16 +++
.../wrappers/streaming/WindowDoFnOperator.java | 18 +++
.../streaming/io/BoundedSourceWrapper.java | 28 +++++
.../streaming/io/UnboundedSourceWrapper.java | 56 ++++-----
12 files changed, 234 insertions(+), 90 deletions(-)
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 11/12: [BEAM-3727] Never shutdown sources in Flink Streaming
execution mode
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 6148a6d063a0503ee435ab5084fcba3fb864b26f
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Feb 22 14:26:16 2018 +0100
[BEAM-3727] Never shutdown sources in Flink Streaming execution mode
This adds an option that controls whether to shutdown sources or not in
case of reaching the +Inf watermark.
The reason for this is https://issues.apache.org/jira/browse/FLINK-2491,
which causes checkpointing to stop once some source is shut down.
---
.../beam/runners/flink/FlinkPipelineOptions.java | 12 +++++
.../apache/beam/runners/flink/TestFlinkRunner.java | 1 +
.../streaming/io/BoundedSourceWrapper.java | 28 +++++++++++
.../streaming/io/UnboundedSourceWrapper.java | 56 ++++++++++++----------
4 files changed, 71 insertions(+), 26 deletions(-)
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
index 01f7847..b2cbefb 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
@@ -137,4 +137,16 @@ public interface FlinkPipelineOptions
Long getMaxBundleTimeMills();
void setMaxBundleTimeMills(Long time);
+ /**
+ * Whether to shutdown sources when their watermark reaches {@code +Inf}. For production use
+ * cases you want this to be disabled because Flink will currently (versions {@literal <=} 1.5)
+ * stop doing checkpoints when any operator (which includes sources) is finished.
+ *
+ * <p>Please see <a href="https://issues.apache.org/jira/browse/FLINK-2491">FLINK-2491</a> for
+ * progress on this issue.
+ */
+ @Description("If set, shutdown sources when their watermark reaches +Inf.")
+ @Default.Boolean(false)
+ Boolean isShutdownSourcesOnFinalWatermark();
+ void setShutdownSourcesOnFinalWatermark(Boolean shutdownOnFinalWatermark);
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
index 01b67e5..47d4494 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
@@ -36,6 +36,7 @@ public class TestFlinkRunner extends PipelineRunner<PipelineResult> {
private TestFlinkRunner(FlinkPipelineOptions options) {
// We use [auto] for testing since this will make it pick up the Testing ExecutionEnvironment
options.setFlinkMaster("[auto]");
+ options.setShutdownSourcesOnFinalWatermark(true);
this.delegate = FlinkRunner.fromOptions(options);
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
index 5ddc46f..6db5426 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
@@ -21,6 +21,7 @@ import com.google.common.annotations.VisibleForTesting;
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.metrics.ReaderInvocationUtil;
import org.apache.beam.sdk.io.BoundedSource;
@@ -180,6 +181,33 @@ public class BoundedSourceWrapper<OutputT>
// emit final Long.MAX_VALUE watermark, just to be sure
ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
+
+ FlinkPipelineOptions options = serializedOptions.get().as(FlinkPipelineOptions.class);
+ if (!options.isShutdownSourcesOnFinalWatermark()) {
+ // do nothing, but still look busy ...
+ // we can't return here since Flink requires that all operators stay up,
+ // otherwise checkpointing would not work correctly anymore
+ //
+ // See https://issues.apache.org/jira/browse/FLINK-2491 for progress on this issue
+
+ // wait until this is canceled
+ final Object waitLock = new Object();
+ while (isRunning) {
+ try {
+ // Flink will interrupt us at some point
+ //noinspection SynchronizationOnLocalVariableOrMethodParameter
+ synchronized (waitLock) {
+ // don't wait indefinitely, in case something goes horribly wrong
+ waitLock.wait(1000);
+ }
+ } catch (InterruptedException e) {
+ if (!isRunning) {
+ // restore the interrupted state, and fall through the loop
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
+ }
}
/**
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
index 817dd74..fc23c01 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
@@ -23,6 +23,7 @@ import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.metrics.ReaderInvocationUtil;
import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
@@ -224,32 +225,7 @@ public class UnboundedSourceWrapper<
serializedOptions.get(),
metricContainer);
- if (localReaders.size() == 0) {
- // do nothing, but still look busy ...
- // also, output a Long.MAX_VALUE watermark since we know that we're not
- // going to emit anything
- // we can't return here since Flink requires that all operators stay up,
- // otherwise checkpointing would not work correctly anymore
- ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
-
- // wait until this is canceled
- final Object waitLock = new Object();
- while (isRunning) {
- try {
- // Flink will interrupt us at some point
- //noinspection SynchronizationOnLocalVariableOrMethodParameter
- synchronized (waitLock) {
- // don't wait indefinitely, in case something goes horribly wrong
- waitLock.wait(1000);
- }
- } catch (InterruptedException e) {
- if (!isRunning) {
- // restore the interrupted state, and fall through the loop
- Thread.currentThread().interrupt();
- }
- }
- }
- } else if (localReaders.size() == 1) {
+ if (localReaders.size() == 1) {
// the easy case, we just read from one reader
UnboundedSource.UnboundedReader<OutputT> reader = localReaders.get(0);
@@ -305,7 +281,35 @@ public class UnboundedSourceWrapper<
hadData = false;
}
}
+ }
+ ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
+
+ FlinkPipelineOptions options = serializedOptions.get().as(FlinkPipelineOptions.class);
+ if (!options.isShutdownSourcesOnFinalWatermark()) {
+ // do nothing, but still look busy ...
+ // we can't return here since Flink requires that all operators stay up,
+ // otherwise checkpointing would not work correctly anymore
+ //
+ // See https://issues.apache.org/jira/browse/FLINK-2491 for progress on this issue
+
+ // wait until this is canceled
+ final Object waitLock = new Object();
+ while (isRunning) {
+ try {
+ // Flink will interrupt us at some point
+ //noinspection SynchronizationOnLocalVariableOrMethodParameter
+ synchronized (waitLock) {
+ // don't wait indefinitely, in case something goes horribly wrong
+ waitLock.wait(1000);
+ }
+ } catch (InterruptedException e) {
+ if (!isRunning) {
+ // restore the interrupted state, and fall through the loop
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
}
}
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 07/12: Make ProcessFnRunner constructor public
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 4ec4644952c05167f52ae0937e043140d26059ae
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Feb 8 13:01:22 2018 +0100
Make ProcessFnRunner constructor public
We need this to be able to instantiate with the constructor in the Flink
Runner.
---
.../src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
index 88275d6..e4dfd13 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
@@ -43,7 +43,7 @@ public class ProcessFnRunner<InputT, OutputT, RestrictionT>
private final Collection<PCollectionView<?>> views;
private final ReadyCheckingSideInputReader sideInputReader;
- ProcessFnRunner(
+ public ProcessFnRunner(
DoFnRunner<KeyedWorkItem<String, KV<InputT, RestrictionT>>, OutputT> underlying,
Collection<PCollectionView<?>> views,
ReadyCheckingSideInputReader sideInputReader) {
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 06/12: [BEAM-2140] Don't use StatefulDoFnRunner when running
SDF in FlinkRunner
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit d15979f2cf0d6bc47d049a6ea157d9d7b2b97848
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Wed Jan 10 06:48:49 2018 +0100
[BEAM-2140] Don't use StatefulDoFnRunner when running SDF in FlinkRunner
---
.../translation/wrappers/streaming/SplittableDoFnOperator.java | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
index 44be5f3..e088b07 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
@@ -26,6 +26,7 @@ import java.util.Map;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
+import org.apache.beam.runners.core.DoFnRunner;
import org.apache.beam.runners.core.KeyedWorkItem;
import org.apache.beam.runners.core.KeyedWorkItems;
import org.apache.beam.runners.core.OutputAndTimeBoundedSplittableProcessElementInvoker;
@@ -88,6 +89,15 @@ public class SplittableDoFnOperator<
}
@Override
+ protected DoFnRunner<
+ KeyedWorkItem<String, KV<InputT, RestrictionT>>, OutputT> createWrappingDoFnRunner(
+ DoFnRunner<KeyedWorkItem<String, KV<InputT, RestrictionT>>, OutputT> wrappedRunner) {
+ // don't wrap in anything because we don't need state cleanup because ProcessFn does
+ // all that
+ return wrappedRunner;
+ }
+
+ @Override
public void open() throws Exception {
super.open();
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 08/12: [BEAM-2140] Use ProcessFnRunner in DoFnRunner for
executing SDF
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 74ac703bbfa40d16e7d1115912768c7a63598d52
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Feb 8 13:02:26 2018 +0100
[BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing SDF
For this to work, we need to also change how we wrap values in
KeyedWorkItems, because ProcessFnRunner expects them to be in the
GlobalWindow.
---
.../flink/FlinkStreamingTransformTranslators.java | 35 ++++++++++++++++++----
.../wrappers/streaming/DoFnOperator.java | 11 +++++--
2 files changed, 39 insertions(+), 7 deletions(-)
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index d39b5c1..f5dc3ce 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -1017,10 +1017,8 @@ class FlinkStreamingTransformTranslators {
WindowedValue.
- FullWindowedValueCoder<SingletonKeyedWorkItem<K, InputT>> windowedWorkItemCoder =
- WindowedValue.getFullCoder(
- workItemCoder,
- input.getWindowingStrategy().getWindowFn().windowCoder());
+ ValueOnlyWindowedValueCoder<SingletonKeyedWorkItem<K, InputT>> windowedWorkItemCoder =
+ WindowedValue.getValueOnlyCoder(workItemCoder);
CoderTypeInformation<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemTypeInfo =
new CoderTypeInformation<>(windowedWorkItemCoder);
@@ -1029,7 +1027,7 @@ class FlinkStreamingTransformTranslators {
DataStream<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemStream =
inputDataStream
- .flatMap(new ToKeyedWorkItem<>())
+ .flatMap(new ToKeyedWorkItemInGlobalWindow<>())
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -1041,6 +1039,33 @@ class FlinkStreamingTransformTranslators {
}
}
+ private static class ToKeyedWorkItemInGlobalWindow<K, InputT>
+ extends RichFlatMapFunction<
+ WindowedValue<KV<K, InputT>>,
+ WindowedValue<SingletonKeyedWorkItem<K, InputT>>> {
+
+ @Override
+ public void flatMap(
+ WindowedValue<KV<K, InputT>> inWithMultipleWindows,
+ Collector<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> out) throws Exception {
+
+ // we need to wrap each one work item per window for now
+ // since otherwise the PushbackSideInputRunner will not correctly
+ // determine whether side inputs are ready
+ //
+ // this is tracked as https://issues.apache.org/jira/browse/BEAM-1850
+ for (WindowedValue<KV<K, InputT>> in : inWithMultipleWindows.explodeWindows()) {
+ SingletonKeyedWorkItem<K, InputT> workItem =
+ new SingletonKeyedWorkItem<>(
+ in.getValue().getKey(),
+ in.withValue(in.getValue().getValue()));
+
+ out.collect(WindowedValue.valueInGlobalWindow(workItem));
+ }
+ }
+ }
+
+
private static class FlattenPCollectionTranslator<T>
extends FlinkStreamingPipelineTranslator.StreamTransformTranslator<
PTransform<PCollection<T>, PCollection<T>>> {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 37f56f5..f9b4ee3 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -39,10 +39,12 @@ import javax.annotation.Nullable;
import org.apache.beam.runners.core.DoFnRunner;
import org.apache.beam.runners.core.DoFnRunners;
import org.apache.beam.runners.core.NullSideInputReader;
+import org.apache.beam.runners.core.ProcessFnRunner;
import org.apache.beam.runners.core.PushbackSideInputDoFnRunner;
import org.apache.beam.runners.core.SideInputHandler;
import org.apache.beam.runners.core.SideInputReader;
import org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner;
+import org.apache.beam.runners.core.SplittableParDoViaKeyedWorkItems;
import org.apache.beam.runners.core.StateInternals;
import org.apache.beam.runners.core.StateNamespace;
import org.apache.beam.runners.core.StateNamespaces;
@@ -352,8 +354,13 @@ public class DoFnOperator<InputT, OutputT>
.scheduleAtFixedRate(
timestamp -> checkInvokeFinishBundleByTime(), bundleCheckPeriod, bundleCheckPeriod);
- pushbackDoFnRunner =
- SimplePushbackSideInputDoFnRunner.create(doFnRunner, sideInputs, sideInputHandler);
+ if (doFn instanceof SplittableParDoViaKeyedWorkItems.ProcessFn) {
+ pushbackDoFnRunner =
+ new ProcessFnRunner<>((DoFnRunner) doFnRunner, sideInputs, sideInputHandler);
+ } else {
+ pushbackDoFnRunner =
+ SimplePushbackSideInputDoFnRunner.create(doFnRunner, sideInputs, sideInputHandler);
+ }
}
@Override
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 09/12: [BEAM-2140] Enable SDF tests for Flink Streaming
Runner
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit c920a39b9781ac604f55059e597e327643d58205
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Jan 5 14:20:19 2018 +0100
[BEAM-2140] Enable SDF tests for Flink Streaming Runner
---
runners/flink/pom.xml | 1 -
1 file changed, 1 deletion(-)
diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index 46917e9..cd4d8c3 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -94,7 +94,6 @@
org.apache.beam.sdk.testing.LargeKeys$Above100MB,
org.apache.beam.sdk.testing.UsesCommittedMetrics,
org.apache.beam.sdk.testing.UsesTestStream,
- org.apache.beam.sdk.testing.UsesSplittableParDo
</excludedGroups>
<parallel>none</parallel>
<failIfNoTests>true</failIfNoTests>
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 10/12: [BEAM-2140] Enable SDF tests in gradle for Flink
Streaming Runner
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 6ba3fc1ff2798764756a7188980facc31eb1ac4c
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Feb 8 13:00:34 2018 +0100
[BEAM-2140] Enable SDF tests in gradle for Flink Streaming Runner
---
runners/flink/build.gradle | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/runners/flink/build.gradle b/runners/flink/build.gradle
index dc77bcf..362288e 100644
--- a/runners/flink/build.gradle
+++ b/runners/flink/build.gradle
@@ -97,13 +97,23 @@ def createValidatesRunnerTask(Map m) {
classpath = configurations.validatesRunner
testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
maxParallelForks 4
- useJUnit {
- includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
- excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
- excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
- excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDo'
- excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+ if (config.streaming) {
+ useJUnit {
+ includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+ excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+ }
+ } else {
+ useJUnit {
+ includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+ excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDo'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+ }
}
}
}
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 03/12: Invoke finishBundle() before teardown() in
DoFnOperator
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 4029e23f3075365eddcdad5187b4fc8a6e590989
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Jan 5 14:22:40 2018 +0100
Invoke finishBundle() before teardown() in DoFnOperator
---
.../runners/flink/translation/wrappers/streaming/DoFnOperator.java | 3 +++
1 file changed, 3 insertions(+)
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 2e7f741..688a7cb 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -363,6 +363,9 @@ public class DoFnOperator<InputT, OutputT>
super.dispose();
checkFinishBundleTimer.cancel(true);
} finally {
+ if (bundleStarted) {
+ invokeFinishBundle();
+ }
doFnInvoker.invokeTeardown();
}
}
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 01/12: Make parameter of
DoFnRunners.lateDataDroppingRunner() more specific
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 50c6d9316c8c5089065dd291b097e06b0c80980e
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Jan 11 10:56:49 2018 +0100
Make parameter of DoFnRunners.lateDataDroppingRunner() more specific
---
.../src/main/java/org/apache/beam/runners/core/DoFnRunners.java | 4 ++--
.../runners/flink/translation/wrappers/streaming/DoFnOperator.java | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
index 9d3e25d..80c830a 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
@@ -80,12 +80,12 @@ public class DoFnRunners {
public static <K, InputT, OutputT, W extends BoundedWindow>
DoFnRunner<KeyedWorkItem<K, InputT>, KV<K, OutputT>> lateDataDroppingRunner(
DoFnRunner<KeyedWorkItem<K, InputT>, KV<K, OutputT>> wrappedRunner,
- StepContext stepContext,
+ TimerInternals timerInternals,
WindowingStrategy<?, W> windowingStrategy) {
return new LateDataDroppingDoFnRunner<>(
wrappedRunner,
windowingStrategy,
- stepContext.timerInternals());
+ timerInternals);
}
/**
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 41a35ce..8ccbd8f 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -314,7 +314,7 @@ public class DoFnOperator<InputT, OutputT>
doFnRunner = DoFnRunners.lateDataDroppingRunner(
(DoFnRunner) doFnRunner,
- stepContext,
+ timerInternals,
windowingStrategy);
} else if (keyCoder != null) {
// It is a stateful DoFn
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 04/12: [BEAM-2140] Ignore event-time timers in
SplittableDoFnOperator
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 2e0b192efbae72fe5c0ef61abfc7b4d1b3bb75f5
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Jan 5 14:16:27 2018 +0100
[BEAM-2140] Ignore event-time timers in SplittableDoFnOperator
SplittableDoFnOperator is only used for executing ProcessFn, which does
not use event-time timers. However, StatefulDoFnRunner does use
event-time timers for state cleanup so this change makes sure that they
don't end up being forwarded to the ProcessFn.
---
.../runners/flink/translation/wrappers/streaming/DoFnOperator.java | 1 -
.../translation/wrappers/streaming/SplittableDoFnOperator.java | 6 ++++++
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 688a7cb..dd2f9c4 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -38,7 +38,6 @@ import java.util.concurrent.ScheduledFuture;
import javax.annotation.Nullable;
import org.apache.beam.runners.core.DoFnRunner;
import org.apache.beam.runners.core.DoFnRunners;
-import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetNewDoFn;
import org.apache.beam.runners.core.NullSideInputReader;
import org.apache.beam.runners.core.PushbackSideInputDoFnRunner;
import org.apache.beam.runners.core.SideInputHandler;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
index 1a418a0..44be5f3 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
@@ -37,6 +37,7 @@ import org.apache.beam.runners.core.TimerInternals;
import org.apache.beam.runners.core.TimerInternalsFactory;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -142,6 +143,11 @@ public class SplittableDoFnOperator<
@Override
public void fireTimer(InternalTimer<?, TimerInternals.TimerData> timer) {
+ if (timer.getNamespace().getDomain().equals(TimeDomain.EVENT_TIME)) {
+ // ignore this, it can only be a state cleanup timers from StatefulDoFnRunner and ProcessFn
+ // does its own state cleanup and should never set event-time timers.
+ return;
+ }
doFnRunner.processElement(
WindowedValue.valueInGlobalWindow(
KeyedWorkItems.timersWorkItem(
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 02/12: Allow overriding DoFnRunners in subclasses of Flink
DoFnOperator
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit c40868fe2fc550ae97c7b2d9308dd8b58b20edab
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Thu Jan 11 10:52:01 2018 +0100
Allow overriding DoFnRunners in subclasses of Flink DoFnOperator
---
.../wrappers/streaming/DoFnOperator.java | 69 +++++++++++-----------
.../wrappers/streaming/WindowDoFnOperator.java | 18 ++++++
2 files changed, 52 insertions(+), 35 deletions(-)
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 8ccbd8f..2e7f741 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -218,6 +218,39 @@ public class DoFnOperator<InputT, OutputT>
return doFn;
}
+ // allow overriding this, for example SplittableDoFnOperator will not create a
+ // stateful DoFn runner because ProcessFn, which is used for executing a Splittable DoFn
+ // doesn't play by the normal DoFn rules and WindowDoFnOperator uses LateDataDroppingDoFnRunner
+ protected DoFnRunner<InputT, OutputT> createWrappingDoFnRunner(
+ DoFnRunner<InputT, OutputT> wrappedRunner) {
+
+ if (keyCoder != null) {
+ StatefulDoFnRunner.CleanupTimer cleanupTimer =
+ new StatefulDoFnRunner.TimeInternalsCleanupTimer(
+ timerInternals, windowingStrategy);
+
+ // we don't know the window type
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ Coder windowCoder = windowingStrategy.getWindowFn().windowCoder();
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ StatefulDoFnRunner.StateCleaner<?> stateCleaner =
+ new StatefulDoFnRunner.StateInternalsStateCleaner<>(
+ doFn, keyedStateInternals, windowCoder);
+
+
+ return DoFnRunners.defaultStatefulDoFnRunner(
+ doFn,
+ wrappedRunner,
+ windowingStrategy,
+ cleanupTimer,
+ stateCleaner);
+
+ } else {
+ return doFnRunner;
+ }
+ }
+
@Override
public void setup(
StreamTask<?, ?> containingTask,
@@ -304,41 +337,7 @@ public class DoFnOperator<InputT, OutputT>
stepContext,
windowingStrategy);
- if (doFn instanceof GroupAlsoByWindowViaWindowSetNewDoFn) {
- // When the doFn is this, we know it came from WindowDoFnOperator and
- // InputT = KeyedWorkItem<K, V>
- // OutputT = KV<K, V>
- //
- // for some K, V
-
-
- doFnRunner = DoFnRunners.lateDataDroppingRunner(
- (DoFnRunner) doFnRunner,
- timerInternals,
- windowingStrategy);
- } else if (keyCoder != null) {
- // It is a stateful DoFn
-
- StatefulDoFnRunner.CleanupTimer cleanupTimer =
- new StatefulDoFnRunner.TimeInternalsCleanupTimer(
- stepContext.timerInternals(), windowingStrategy);
-
- // we don't know the window type
- @SuppressWarnings({"unchecked", "rawtypes"})
- Coder windowCoder = windowingStrategy.getWindowFn().windowCoder();
-
- @SuppressWarnings({"unchecked", "rawtypes"})
- StatefulDoFnRunner.StateCleaner<?> stateCleaner =
- new StatefulDoFnRunner.StateInternalsStateCleaner<>(
- doFn, stepContext.stateInternals(), windowCoder);
-
- doFnRunner = DoFnRunners.defaultStatefulDoFnRunner(
- doFn,
- doFnRunner,
- windowingStrategy,
- cleanupTimer,
- stateCleaner);
- }
+ doFnRunner = createWrappingDoFnRunner(doFnRunner);
if (options.getEnableMetrics()) {
doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
index 7a04238..8447ade 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
@@ -23,6 +23,8 @@ import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
+import org.apache.beam.runners.core.DoFnRunner;
+import org.apache.beam.runners.core.DoFnRunners;
import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetNewDoFn;
import org.apache.beam.runners.core.KeyedWorkItem;
import org.apache.beam.runners.core.KeyedWorkItems;
@@ -79,6 +81,22 @@ public class WindowDoFnOperator<K, InputT, OutputT>
}
@Override
+ protected DoFnRunner<KeyedWorkItem<K, InputT>, KV<K, OutputT>> createWrappingDoFnRunner(
+ DoFnRunner<KeyedWorkItem<K, InputT>, KV<K, OutputT>> wrappedRunner) {
+ // When the doFn is this, we know it came from WindowDoFnOperator and
+ // InputT = KeyedWorkItem<K, V>
+ // OutputT = KV<K, V>
+ //
+ // for some K, V
+
+
+ return DoFnRunners.lateDataDroppingRunner(
+ (DoFnRunner) doFnRunner,
+ timerInternals,
+ windowingStrategy);
+ }
+
+ @Override
protected DoFn<KeyedWorkItem<K, InputT>, KV<K, OutputT>> getDoFn() {
StateInternalsFactory<K> stateInternalsFactory =
key -> {
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.
[beam] 05/12: [BEAM-2140] Block DoFnOperator.close() if we have
pending timers
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 3eecb381f3fb412df6844cfc13b12bf265253926
Author: Aljoscha Krettek <al...@gmail.com>
AuthorDate: Fri Jan 5 14:17:49 2018 +0100
[BEAM-2140] Block DoFnOperator.close() if we have pending timers
It can happen that the input operation finishes while we still have
pending processing-time timers, for example from processing a Splittable
DoFn. This change makes sure that we block as long as we have pending
timers.
This change also makes sure that we forward a +Inf watermark in close().
We have to do this because it can happen that we get a +Inf watermark on
input while we still have active watermark holds (which will get
resolved when all pending timers are gone). With this change we make
sure to send a +Inf watermark downstream once everything is resolved.
---
.../wrappers/streaming/DoFnOperator.java | 43 +++++++++++++++++-----
1 file changed, 33 insertions(+), 10 deletions(-)
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index dd2f9c4..37f56f5 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -371,18 +371,41 @@ public class DoFnOperator<InputT, OutputT>
@Override
public void close() throws Exception {
- super.close();
-
- // sanity check: these should have been flushed out by +Inf watermarks
- if (!sideInputs.isEmpty() && nonKeyedStateInternals != null) {
- BagState<WindowedValue<InputT>> pushedBack =
- nonKeyedStateInternals.state(StateNamespaces.global(), pushedBackTag);
+ try {
- Iterable<WindowedValue<InputT>> pushedBackContents = pushedBack.read();
- if (pushedBackContents != null && !Iterables.isEmpty(pushedBackContents)) {
- String pushedBackString = Joiner.on(",").join(pushedBackContents);
+ // This is our last change to block shutdown of this operator while
+ // there are still remaining processing-time timers. Flink will ignore pending
+ // processing-time timers when upstream operators have shut down and will also
+ // shut down this operator with pending processing-time timers.
+ while (this.numProcessingTimeTimers() > 0) {
+ getContainingTask().getCheckpointLock().wait(100);
+ }
+ if (this.numProcessingTimeTimers() > 0) {
throw new RuntimeException(
- "Leftover pushed-back data: " + pushedBackString + ". This indicates a bug.");
+ "There are still processing-time timers left, this indicates a bug");
+ }
+
+ // make sure we send a +Inf watermark downstream. It can happen that we receive +Inf
+ // in processWatermark*() but have holds, so we have to re-evaluate here.
+ processWatermark(new Watermark(Long.MAX_VALUE));
+ if (currentOutputWatermark < Long.MAX_VALUE) {
+ throw new RuntimeException("There are still watermark holds. Watermark held at "
+ + keyedStateInternals.watermarkHold().getMillis() + ".");
+ }
+ } finally {
+ super.close();
+
+ // sanity check: these should have been flushed out by +Inf watermarks
+ if (!sideInputs.isEmpty() && nonKeyedStateInternals != null) {
+ BagState<WindowedValue<InputT>> pushedBack =
+ nonKeyedStateInternals.state(StateNamespaces.global(), pushedBackTag);
+
+ Iterable<WindowedValue<InputT>> pushedBackContents = pushedBack.read();
+ if (pushedBackContents != null && !Iterables.isEmpty(pushedBackContents)) {
+ String pushedBackString = Joiner.on(",").join(pushedBackContents);
+ throw new RuntimeException(
+ "Leftover pushed-back data: " + pushedBackString + ". This indicates a bug.");
+ }
}
}
}
--
To stop receiving notification emails like this one, please contact
aljoscha@apache.org.