You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by th...@apache.org on 2016/11/07 19:58:51 UTC
[01/50] incubator-beam git commit: This closes #1170
Repository: incubator-beam
Updated Branches:
refs/heads/apex-runner 15287b838 -> 9197d1e05
This closes #1170
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/529f266a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/529f266a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/529f266a
Branch: refs/heads/apex-runner
Commit: 529f266ae91167997a17a952ee98b1721bba47a8
Parents: 2c0d0f4 ff5409f
Author: Sela <an...@paypal.com>
Authored: Wed Nov 2 14:54:21 2016 +0200
Committer: Sela <an...@paypal.com>
Committed: Wed Nov 2 14:54:21 2016 +0200
----------------------------------------------------------------------
runners/spark/pom.xml | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
[16/50] incubator-beam git commit: This closes #1154
Posted by th...@apache.org.
This closes #1154
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/eac68cb2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/eac68cb2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/eac68cb2
Branch: refs/heads/apex-runner
Commit: eac68cb217bb635649f900f57333f7fab40e77a7
Parents: 55d9519 0cb3832
Author: Thomas Groh <tg...@google.com>
Authored: Thu Nov 3 13:45:34 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Thu Nov 3 13:45:34 2016 -0700
----------------------------------------------------------------------
.../apache/beam/runners/direct/EvaluationContext.java | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
[41/50] incubator-beam git commit: Closes #1290
Posted by th...@apache.org.
Closes #1290
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5864a38b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5864a38b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5864a38b
Branch: refs/heads/apex-runner
Commit: 5864a38ba595cdf90d2d5559e4312ef6144f60c7
Parents: 14e093a 717b431
Author: Dan Halperin <dh...@google.com>
Authored: Sat Nov 5 01:30:41 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Sat Nov 5 01:30:41 2016 -0700
----------------------------------------------------------------------
examples/java/pom.xml | 68 -------------------
examples/java8/pom.xml | 8 ---
.../beam/examples/complete/game/GameStats.java | 3 +-
.../examples/complete/game/LeaderBoard.java | 2 +-
.../beam/examples/complete/game/UserScore.java | 2 +-
pom.xml | 11 ++-
runners/core-java/pom.xml | 23 -------
runners/direct-java/pom.xml | 61 -----------------
runners/flink/examples/pom.xml | 8 ---
.../flink/examples/streaming/AutoComplete.java | 8 +--
.../flink/examples/streaming/JoinExamples.java | 12 ++--
.../examples/streaming/KafkaIOExamples.java | 2 +-
.../examples/streaming/WindowedWordCount.java | 8 +--
runners/flink/runner/pom.xml | 8 ---
runners/google-cloud-dataflow-java/pom.xml | 70 --------------------
.../runners/dataflow/DataflowPipelineJob.java | 1 -
.../runners/dataflow/internal/IsmFormat.java | 5 +-
.../runners/dataflow/util/MonitoringUtil.java | 1 -
.../runners/dataflow/util/RandomAccessData.java | 2 +-
runners/spark/pom.xml | 4 --
sdks/java/core/pom.xml | 70 --------------------
.../java/org/apache/beam/sdk/io/AvroIO.java | 4 +-
.../org/apache/beam/sdk/io/BoundedSource.java | 11 ++-
.../apache/beam/sdk/io/CompressedSource.java | 2 -
.../org/apache/beam/sdk/io/FileBasedSink.java | 9 +--
.../java/org/apache/beam/sdk/io/XmlSource.java | 2 +-
.../apache/beam/sdk/io/range/ByteKeyRange.java | 2 +-
.../apache/beam/sdk/options/ValueProvider.java | 2 +-
.../apache/beam/sdk/transforms/Aggregator.java | 12 ++--
.../apache/beam/sdk/transforms/CombineFns.java | 4 +-
.../org/apache/beam/sdk/transforms/DoFn.java | 22 +++---
.../org/apache/beam/sdk/transforms/ParDo.java | 2 +-
.../org/apache/beam/sdk/transforms/View.java | 2 +-
.../sdk/transforms/display/DisplayData.java | 5 +-
.../beam/sdk/transforms/join/CoGroupByKey.java | 6 +-
.../sdk/transforms/reflect/DoFnSignature.java | 2 +-
.../splittabledofn/RestrictionTracker.java | 2 +-
.../transforms/splittabledofn/package-info.java | 4 +-
.../beam/sdk/util/BaseExecutionContext.java | 2 +-
.../BufferedElementCountingOutputStream.java | 4 +-
.../apache/beam/sdk/util/ExecutionContext.java | 8 +--
.../sdk/util/ExposedByteArrayInputStream.java | 5 +-
.../sdk/util/ExposedByteArrayOutputStream.java | 2 -
.../java/org/apache/beam/sdk/util/GcsUtil.java | 2 -
.../apache/beam/sdk/util/MovingFunction.java | 4 +-
.../org/apache/beam/sdk/util/PubsubClient.java | 32 ++-------
.../org/apache/beam/sdk/util/StringUtils.java | 8 +--
.../apache/beam/sdk/util/TimerInternals.java | 8 ++-
.../beam/sdk/util/state/StateNamespace.java | 16 ++---
.../java/org/apache/beam/sdk/values/PDone.java | 2 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 2 +-
sdks/java/io/jdbc/pom.xml | 4 --
sdks/java/io/kafka/pom.xml | 7 --
sdks/java/io/kinesis/pom.xml | 7 --
.../apache/beam/sdk/io/kinesis/KinesisIO.java | 35 +++++-----
sdks/java/io/mongodb/pom.xml | 4 --
56 files changed, 119 insertions(+), 503 deletions(-)
----------------------------------------------------------------------
[29/50] incubator-beam git commit: checkstyle: move from individual
modules to root poms
Posted by th...@apache.org.
checkstyle: move from individual modules to root poms
Checkstyle config is no longer needed in individual modules;
instead, move it only to sdks/ , runners/, and examples/ poms.
Individual modules like sdks/java/core may still reference checkstyle
to configure their own options.
The only sketchy bit is that I had to link sdks/java/build-tools to
beam-parent rather than sdks/java so that I didn't create a build
dependency loop.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f50b2142
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f50b2142
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f50b2142
Branch: refs/heads/apex-runner
Commit: f50b2142b641c6d2aca186903139cff995f7d49e
Parents: faf55c7
Author: Dan Halperin <dh...@google.com>
Authored: Thu Nov 3 17:24:52 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Fri Nov 4 10:51:28 2016 -0700
----------------------------------------------------------------------
examples/java/pom.xml | 5 -----
examples/java8/pom.xml | 5 -----
examples/pom.xml | 13 ++++++++++++-
pom.xml | 3 +++
runners/core-java/pom.xml | 5 -----
runners/direct-java/pom.xml | 5 -----
runners/flink/examples/pom.xml | 5 -----
runners/flink/runner/pom.xml | 5 -----
runners/google-cloud-dataflow-java/pom.xml | 7 -------
runners/pom.xml | 11 +++++++++++
runners/spark/pom.xml | 4 ----
sdks/java/build-tools/pom.xml | 4 ++--
.../src/main/resources/beam/checkstyle.xml | 2 +-
sdks/java/core/pom.xml | 18 +++++++++---------
sdks/java/extensions/join-library/pom.xml | 4 ----
sdks/java/extensions/sorter/pom.xml | 4 ----
sdks/java/io/google-cloud-platform/pom.xml | 4 ----
sdks/java/io/hdfs/pom.xml | 4 ----
sdks/java/io/jdbc/pom.xml | 6 +-----
sdks/java/io/jms/pom.xml | 4 ----
sdks/java/io/kafka/pom.xml | 4 ----
sdks/java/io/kinesis/pom.xml | 4 ----
sdks/java/io/mongodb/pom.xml | 4 ----
sdks/java/java8tests/pom.xml | 5 -----
sdks/java/microbenchmarks/pom.xml | 5 -----
sdks/java/pom.xml | 4 +++-
sdks/pom.xml | 16 +++++++++++++++-
27 files changed, 57 insertions(+), 103 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/examples/java/pom.xml
----------------------------------------------------------------------
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index fc82ed4..12a114f 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -277,11 +277,6 @@
</configuration>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
<!-- Source plugin for generating source and test-source JARs. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/examples/java8/pom.xml
----------------------------------------------------------------------
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index e6408dc..18257d9 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -148,11 +148,6 @@
</configuration>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
<!-- Source plugin for generating source and test-source JARs. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/examples/pom.xml
----------------------------------------------------------------------
diff --git a/examples/pom.xml b/examples/pom.xml
index 2820473..c6f9cb3 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -45,7 +45,18 @@
<modules>
<module>java8</module>
</modules>
- </profile>
+ </profile>
+ <profile>
+ <id>release</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
</profiles>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index ea7d4ae..28845a5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -140,6 +140,9 @@
<packaging>pom</packaging>
<modules>
+ <!-- sdks/java/build-tools has project-wide configuration. To make these available
+ in all modules, link it directly to the parent pom.xml. -->
+ <module>sdks/java/build-tools</module>
<module>sdks</module>
<module>runners</module>
<!-- sdks/java/maven-archetypes has several dependencies on the DataflowPipelineRunner.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/runners/core-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index e3e4fed..fa64660 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -54,11 +54,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/runners/direct-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index 84190b2..b781c6e 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -48,11 +48,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/runners/flink/examples/pom.xml
----------------------------------------------------------------------
diff --git a/runners/flink/examples/pom.xml b/runners/flink/examples/pom.xml
index a0cf676..409d77b 100644
--- a/runners/flink/examples/pom.xml
+++ b/runners/flink/examples/pom.xml
@@ -111,11 +111,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<configuration>
<additionalparam>-Xdoclint:missing</additionalparam>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/runners/flink/runner/pom.xml
----------------------------------------------------------------------
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 283d060..928889e 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -244,11 +244,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<configuration>
<additionalparam>-Xdoclint:missing</additionalparam>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/runners/google-cloud-dataflow-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/google-cloud-dataflow-java/pom.xml b/runners/google-cloud-dataflow-java/pom.xml
index 750f520..d8f1b82 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -72,13 +72,6 @@
</executions>
</plugin>
- <!-- Run CheckStyle pass on transforms, as they are release in
- source form. -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/runners/pom.xml
----------------------------------------------------------------------
diff --git a/runners/pom.xml b/runners/pom.xml
index b2b613f..9c821cc 100644
--- a/runners/pom.xml
+++ b/runners/pom.xml
@@ -41,6 +41,17 @@
</modules>
<profiles>
+ <profile>
+ <id>release</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
<!-- A profile that adds an integration test phase if and only if
the runnableOnServicePipelineOptions maven property has been set.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/runners/spark/pom.xml
----------------------------------------------------------------------
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 71a3ac2..0edafe8 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -411,10 +411,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/build-tools/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/build-tools/pom.xml b/sdks/java/build-tools/pom.xml
index cc27bea..b911328 100644
--- a/sdks/java/build-tools/pom.xml
+++ b/sdks/java/build-tools/pom.xml
@@ -21,9 +21,9 @@
<parent>
<groupId>org.apache.beam</groupId>
- <artifactId>beam-sdks-java-parent</artifactId>
+ <artifactId>beam-parent</artifactId>
<version>0.4.0-incubating-SNAPSHOT</version>
- <relativePath>../pom.xml</relativePath>
+ <relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>beam-sdks-java-build-tools</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
----------------------------------------------------------------------
diff --git a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
index e57cfec..ebbaa7d 100644
--- a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
@@ -42,7 +42,7 @@ page at http://checkstyle.sourceforge.net/config.html -->
</module>
<module name="RegexpSingleline">
- <property name="format" value="[ \t]+$"/>
+ <property name="format" value="\s+$"/>
<property name="message" value="Trailing whitespace"/>
<property name="severity" value="error"/>
</module>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/core/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index 29aaaab..40a511c 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -106,6 +106,15 @@
</offlineLinks>
</configuration>
</plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ <configuration>
+ <!-- Set testSourceDirectory in order to exclude generated-test-sources -->
+ <testSourceDirectory>${project.basedir}/src/test/</testSourceDirectory>
+ </configuration>
+ </plugin>
</plugins>
</pluginManagement>
@@ -130,15 +139,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- <configuration>
- <!-- Set testSourceDirectory in order to exclude generated-test-sources -->
- <testSourceDirectory>${project.basedir}/src/test/</testSourceDirectory>
- </configuration>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/extensions/join-library/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/extensions/join-library/pom.xml b/sdks/java/extensions/join-library/pom.xml
index 4687554..16326b0 100644
--- a/sdks/java/extensions/join-library/pom.xml
+++ b/sdks/java/extensions/join-library/pom.xml
@@ -46,10 +46,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/extensions/sorter/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/extensions/sorter/pom.xml b/sdks/java/extensions/sorter/pom.xml
index e77d5ad..1a994de 100644
--- a/sdks/java/extensions/sorter/pom.xml
+++ b/sdks/java/extensions/sorter/pom.xml
@@ -50,10 +50,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/io/google-cloud-platform/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/google-cloud-platform/pom.xml b/sdks/java/io/google-cloud-platform/pom.xml
index 054ee21..fa51072 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -58,10 +58,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
<!-- Integration Tests -->
<plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/io/hdfs/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/hdfs/pom.xml b/sdks/java/io/hdfs/pom.xml
index 22c3187..e79996f 100644
--- a/sdks/java/io/hdfs/pom.xml
+++ b/sdks/java/io/hdfs/pom.xml
@@ -53,10 +53,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
</plugins>
</build>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/io/jdbc/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/pom.xml b/sdks/java/io/jdbc/pom.xml
index 3f513ca..a65ed46 100644
--- a/sdks/java/io/jdbc/pom.xml
+++ b/sdks/java/io/jdbc/pom.xml
@@ -50,10 +50,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
</plugin>
</plugins>
@@ -135,4 +131,4 @@
</dependency>
</dependencies>
-</project>
\ No newline at end of file
+</project>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/io/jms/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/jms/pom.xml b/sdks/java/io/jms/pom.xml
index 5a74b34..0b46154 100644
--- a/sdks/java/io/jms/pom.xml
+++ b/sdks/java/io/jms/pom.xml
@@ -50,10 +50,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
</plugins>
</build>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/io/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/kafka/pom.xml b/sdks/java/io/kafka/pom.xml
index f2b8326..184c1c9 100644
--- a/sdks/java/io/kafka/pom.xml
+++ b/sdks/java/io/kafka/pom.xml
@@ -54,10 +54,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<configuration>
<additionalparam>-Xdoclint:missing</additionalparam>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/io/kinesis/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/kinesis/pom.xml b/sdks/java/io/kinesis/pom.xml
index 0f65d67..76df329 100644
--- a/sdks/java/io/kinesis/pom.xml
+++ b/sdks/java/io/kinesis/pom.xml
@@ -54,10 +54,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<configuration>
<additionalparam>-Xdoclint:missing</additionalparam>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/io/mongodb/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/mongodb/pom.xml b/sdks/java/io/mongodb/pom.xml
index ad0609d..d674e63 100644
--- a/sdks/java/io/mongodb/pom.xml
+++ b/sdks/java/io/mongodb/pom.xml
@@ -50,10 +50,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
</plugin>
</plugins>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/java8tests/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/java8tests/pom.xml b/sdks/java/java8tests/pom.xml
index 13f5fce..989fba2 100644
--- a/sdks/java/java8tests/pom.xml
+++ b/sdks/java/java8tests/pom.xml
@@ -61,11 +61,6 @@
</configuration>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
<!-- Source plugin for generating source and test-source JARs. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/microbenchmarks/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/microbenchmarks/pom.xml b/sdks/java/microbenchmarks/pom.xml
index 5c6cf4d..05f06aa 100644
--- a/sdks/java/microbenchmarks/pom.xml
+++ b/sdks/java/microbenchmarks/pom.xml
@@ -50,11 +50,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/java/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/pom.xml b/sdks/java/pom.xml
index eae5a58..2866cb3 100644
--- a/sdks/java/pom.xml
+++ b/sdks/java/pom.xml
@@ -33,7 +33,9 @@
<name>Apache Beam :: SDKs :: Java</name>
<modules>
- <module>build-tools</module>
+ <!-- build-tools inherits from the Beam root pom to enable checkstyle
+ and other project configuration to be used in all modules.
+ <module>build-tools</module> -->
<module>core</module>
<module>io</module>
<!-- sdks/java/maven-archtypes has several dependencies on the
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f50b2142/sdks/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/pom.xml b/sdks/pom.xml
index 235d102..6347fe1 100644
--- a/sdks/pom.xml
+++ b/sdks/pom.xml
@@ -36,6 +36,20 @@
<module>java</module>
</modules>
+ <profiles>
+ <profile>
+ <id>release</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
<build>
<pluginManagement>
<plugins>
@@ -57,4 +71,4 @@
</pluginManagement>
</build>
-</project>
\ No newline at end of file
+</project>
[15/50] incubator-beam git commit: Closes #1050
Posted by th...@apache.org.
Closes #1050
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/55d95198
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/55d95198
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/55d95198
Branch: refs/heads/apex-runner
Commit: 55d951987e79b6c9c97d14ea72d60293a3537b9d
Parents: c6c41ea 8c1008c
Author: Dan Halperin <dh...@google.com>
Authored: Thu Nov 3 12:39:13 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Thu Nov 3 12:39:13 2016 -0700
----------------------------------------------------------------------
.../examples/MinimalWordCountJava8Test.java | 2 +-
.../org/apache/beam/sdk/io/FileBasedSink.java | 34 +++++++++++++++-----
.../beam/sdk/util/FileIOChannelFactory.java | 8 +++--
.../beam/sdk/util/GcsIOChannelFactory.java | 8 ++++-
.../apache/beam/sdk/util/IOChannelFactory.java | 4 +++
.../org/apache/beam/sdk/util/gcsfs/GcsPath.java | 13 ++++++--
.../apache/beam/sdk/io/FileBasedSinkTest.java | 23 +++++++------
.../org/apache/beam/sdk/io/XmlSinkTest.java | 16 +++++++--
.../apache/beam/sdk/util/gcsfs/GcsPathTest.java | 25 ++++++++++++++
9 files changed, 107 insertions(+), 26 deletions(-)
----------------------------------------------------------------------
[04/50] incubator-beam git commit: [BEAM-887] Remove false/crashing
"backwards compatibility" from Sink
Posted by th...@apache.org.
[BEAM-887] Remove false/crashing "backwards compatibility" from Sink
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2f6104c9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2f6104c9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2f6104c9
Branch: refs/heads/apex-runner
Commit: 2f6104c903395ac8444fcaab8830e7178ebfe050
Parents: 5224693
Author: Kenneth Knowles <kl...@google.com>
Authored: Wed Nov 2 11:56:43 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Wed Nov 2 15:53:19 2016 -0700
----------------------------------------------------------------------
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Sink.java | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2f6104c9/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Sink.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Sink.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Sink.java
index f256c5f..d1770c8 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Sink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Sink.java
@@ -214,9 +214,7 @@ public abstract class Sink<T> implements Serializable, HasDisplayData {
/**
* Returns a coder for the writer result type.
*/
- public Coder<WriteT> getWriterResultCoder() {
- return null;
- }
+ public abstract Coder<WriteT> getWriterResultCoder();
}
/**
[50/50] incubator-beam git commit: Adjust for merge from master.
Posted by th...@apache.org.
Adjust for merge from master.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9197d1e0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9197d1e0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9197d1e0
Branch: refs/heads/apex-runner
Commit: 9197d1e05e034f8dea0b413c09e7428e90294509
Parents: 5ccd5fb
Author: Thomas Weise <th...@apache.org>
Authored: Mon Nov 7 20:29:08 2016 +0100
Committer: Thomas Weise <th...@apache.org>
Committed: Mon Nov 7 20:29:08 2016 +0100
----------------------------------------------------------------------
.../translators/functions/ApexGroupByKeyOperator.java | 14 ++++++++++++++
1 file changed, 14 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9197d1e0/runners/apex/src/main/java/org/apache/beam/runners/apex/translators/functions/ApexGroupByKeyOperator.java
----------------------------------------------------------------------
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translators/functions/ApexGroupByKeyOperator.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translators/functions/ApexGroupByKeyOperator.java
index 98f3eca..4c28c85 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translators/functions/ApexGroupByKeyOperator.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translators/functions/ApexGroupByKeyOperator.java
@@ -60,12 +60,14 @@ import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.KeyedWorkItem;
import org.apache.beam.sdk.util.KeyedWorkItems;
+import org.apache.beam.sdk.util.TimeDomain;
import org.apache.beam.sdk.util.TimerInternals;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowingInternals;
import org.apache.beam.sdk.util.WindowingStrategy;
import org.apache.beam.sdk.util.state.StateInternals;
import org.apache.beam.sdk.util.state.StateInternalsFactory;
+import org.apache.beam.sdk.util.state.StateNamespace;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
@@ -451,6 +453,18 @@ public class ApexGroupByKeyOperator<K, V> implements Operator {
// TODO Auto-generated method stub
return null;
}
+
+ @Override
+ public void setTimer(StateNamespace namespace, String timerId, Instant target,
+ TimeDomain timeDomain) {
+ throw new UnsupportedOperationException("Setting timer by ID not yet supported.");
+ }
+
+ @Override
+ public void deleteTimer(StateNamespace namespace, String timerId) {
+ throw new UnsupportedOperationException("Canceling of timer by ID is not yet supported.");
+ }
+
}
private class GroupByKeyStateInternalsFactory implements StateInternalsFactory<K>, Serializable {
[33/50] incubator-beam git commit: This closes #1276
Posted by th...@apache.org.
This closes #1276
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/99062d10
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/99062d10
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/99062d10
Branch: refs/heads/apex-runner
Commit: 99062d103c96e47ea78fc866984195df12de7770
Parents: 6e1e57b 5dca267
Author: Thomas Groh <tg...@google.com>
Authored: Fri Nov 4 13:05:21 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Fri Nov 4 13:05:21 2016 -0700
----------------------------------------------------------------------
.../beam/runners/direct/EvaluationContext.java | 6 +-
.../direct/ExecutorServiceParallelExecutor.java | 41 ++++----
.../beam/runners/direct/WatermarkManager.java | 79 ++++++++------
.../runners/direct/EvaluationContextTest.java | 23 ++---
.../runners/direct/WatermarkManagerTest.java | 102 ++++++-------------
5 files changed, 109 insertions(+), 142 deletions(-)
----------------------------------------------------------------------
[38/50] incubator-beam git commit: This closes #1272
Posted by th...@apache.org.
This closes #1272
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/14e093a0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/14e093a0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/14e093a0
Branch: refs/heads/apex-runner
Commit: 14e093a0a574c8c3920a83c38e411a06b29bf44b
Parents: 46fbfe0 90a75d1
Author: Sela <an...@paypal.com>
Authored: Sat Nov 5 00:02:22 2016 +0200
Committer: Sela <an...@paypal.com>
Committed: Sat Nov 5 00:02:22 2016 +0200
----------------------------------------------------------------------
.../runners/spark/SparkPipelineOptions.java | 3 +--
.../SparkRunnerStreamingContextFactory.java | 23 +++++---------------
.../streaming/EmptyStreamAssertionTest.java | 3 +--
.../streaming/FlattenStreamingTest.java | 6 ++---
.../streaming/KafkaStreamingTest.java | 6 ++---
.../ResumeFromCheckpointStreamingTest.java | 3 +--
.../streaming/SimpleStreamingWordCountTest.java | 3 +--
.../utils/TestOptionsForStreaming.java | 12 +++++-----
8 files changed, 19 insertions(+), 40 deletions(-)
----------------------------------------------------------------------
[26/50] incubator-beam git commit: Add TypeDescriptor#getTypes
Posted by th...@apache.org.
Add TypeDescriptor#getTypes
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d936ed89
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d936ed89
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d936ed89
Branch: refs/heads/apex-runner
Commit: d936ed896be4951bfd8766906b214af98a000f34
Parents: 3419fef
Author: Kenneth Knowles <kl...@google.com>
Authored: Tue Nov 1 15:38:01 2016 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Thu Nov 3 21:32:53 2016 -0700
----------------------------------------------------------------------
.../org/apache/beam/sdk/values/TypeDescriptor.java | 13 +++++++++++++
1 file changed, 13 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d936ed89/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
index 6eabf42..14f2cb8 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
@@ -289,6 +289,19 @@ public abstract class TypeDescriptor<T> implements Serializable {
}
/**
+ * Returns a set of {@link TypeDescriptor TypeDescriptor}, one for each
+ * superclass as well as each interface implemented by this class.
+ */
+ @SuppressWarnings("rawtypes")
+ public Iterable<TypeDescriptor> getTypes() {
+ List<TypeDescriptor> interfaces = Lists.newArrayList();
+ for (TypeToken<?> interfaceToken : token.getTypes()) {
+ interfaces.add(new SimpleTypeDescriptor<>(interfaceToken));
+ }
+ return interfaces;
+ }
+
+ /**
* Returns a set of {@link TypeDescriptor}s, one for each
* interface implemented by this class.
*/
[22/50] incubator-beam git commit: Add Tests for
PAssert.IterableAssert#empty()
Posted by th...@apache.org.
Add Tests for PAssert.IterableAssert#empty()
These tests demonstrate that an empty PCollection has no elements, and
that the empty() assertion affirmatively fails when there are elements
within the PCollection.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e0764767
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e0764767
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e0764767
Branch: refs/heads/apex-runner
Commit: e0764767e1806b3ac82064ecc108c3944bfe8a22
Parents: b93ac70
Author: Thomas Groh <tg...@google.com>
Authored: Wed Nov 2 10:35:27 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Thu Nov 3 15:33:43 2016 -0700
----------------------------------------------------------------------
.../apache/beam/sdk/testing/PAssertTest.java | 27 ++++++++++++++++++++
1 file changed, 27 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e0764767/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
index 54ddd3f..be8924f 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.testing;
+import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
@@ -32,6 +33,8 @@ import java.util.regex.Pattern;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.io.CountingInput;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
@@ -355,6 +358,18 @@ public class PAssertTest implements Serializable {
pipeline.run();
}
+ @Test
+ @Category(RunnableOnService.class)
+ public void testEmpty() {
+ Pipeline p = TestPipeline.create();
+ PCollection<Long> vals =
+ p.apply(Create.<Long>of().withCoder(VarLongCoder.of()));
+
+ PAssert.that(vals).empty();
+
+ p.run();
+ }
+
/**
* Tests that {@code containsInAnyOrder} fails when and how it should.
*/
@@ -381,6 +396,18 @@ public class PAssertTest implements Serializable {
expectedPattern.matcher(exc.getMessage()).find());
}
+ @Test
+ @Category(RunnableOnService.class)
+ public void testEmptyFalse() throws Exception {
+ Pipeline p = TestPipeline.create();
+ PCollection<Long> vals = p.apply(CountingInput.upTo(5L));
+ PAssert.that(vals).empty();
+
+ Throwable thrown = runExpectingAssertionFailure(p);
+
+ assertThat(thrown.getMessage(), containsString("Expected: iterable over [] in any order"));
+ }
+
private static Throwable runExpectingAssertionFailure(Pipeline pipeline) {
// We cannot use thrown.expect(AssertionError.class) because the AssertionError
// is first caught by JUnit and causes a test failure.
[23/50] incubator-beam git commit: This closes #1258
Posted by th...@apache.org.
This closes #1258
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3419fef7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3419fef7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3419fef7
Branch: refs/heads/apex-runner
Commit: 3419fef7f2203840dfd938d104db7d25540b4ff3
Parents: b93ac70 e076476
Author: Thomas Groh <tg...@google.com>
Authored: Thu Nov 3 15:33:44 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Thu Nov 3 15:33:44 2016 -0700
----------------------------------------------------------------------
.../apache/beam/sdk/testing/PAssertTest.java | 27 ++++++++++++++++++++
1 file changed, 27 insertions(+)
----------------------------------------------------------------------
[14/50] incubator-beam git commit: Makes FileBasedSink use a
temporary directory
Posted by th...@apache.org.
Makes FileBasedSink use a temporary directory
When writing to /path/to/foo, temporary files would be
written to /path/too/foo-temp-$uid (or something like that),
i.e. as siblings of the final output. That could lead
to issues like http://stackoverflow.com/q/39822859/278042
Now, temporary files are written to a path like:
/path/too/temp-beam-foo-$date/$uid.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8c1008c4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8c1008c4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8c1008c4
Branch: refs/heads/apex-runner
Commit: 8c1008c42615dd7704fc2df02b140af1c8332c72
Parents: c6c41ea
Author: Eugene Kirpichov <ki...@google.com>
Authored: Tue Oct 4 15:23:27 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Thu Nov 3 12:39:12 2016 -0700
----------------------------------------------------------------------
.../examples/MinimalWordCountJava8Test.java | 2 +-
.../org/apache/beam/sdk/io/FileBasedSink.java | 34 +++++++++++++++-----
.../beam/sdk/util/FileIOChannelFactory.java | 8 +++--
.../beam/sdk/util/GcsIOChannelFactory.java | 8 ++++-
.../apache/beam/sdk/util/IOChannelFactory.java | 4 +++
.../org/apache/beam/sdk/util/gcsfs/GcsPath.java | 13 ++++++--
.../apache/beam/sdk/io/FileBasedSinkTest.java | 23 +++++++------
.../org/apache/beam/sdk/io/XmlSinkTest.java | 16 +++++++--
.../apache/beam/sdk/util/gcsfs/GcsPathTest.java | 25 ++++++++++++++
9 files changed, 107 insertions(+), 26 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java
----------------------------------------------------------------------
diff --git a/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java b/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java
index 1819219..f373343 100644
--- a/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java
+++ b/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java
@@ -68,7 +68,7 @@ public class MinimalWordCountJava8Test implements Serializable {
.apply(MapElements
.via((KV<String, Long> wordCount) -> wordCount.getKey() + ": " + wordCount.getValue())
.withOutputType(TypeDescriptors.strings()))
- .apply(TextIO.Write.to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
+ .apply(TextIO.Write.to("gs://your-output-bucket/and-output-prefix"));
}
private GcsUtil buildMockGcsUtil() throws IOException {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index 4355962..3d0fe04 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -31,6 +31,7 @@ import java.nio.channels.Channels;
import java.nio.channels.WritableByteChannel;
import java.nio.file.Files;
import java.nio.file.NoSuchFileException;
+import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.ArrayList;
@@ -53,6 +54,8 @@ import org.apache.beam.sdk.util.IOChannelFactory;
import org.apache.beam.sdk.util.IOChannelUtils;
import org.apache.beam.sdk.util.MimeTypes;
import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream;
+import org.joda.time.Instant;
+import org.joda.time.format.DateTimeFormat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -317,17 +320,16 @@ public abstract class FileBasedSink<T> extends Sink<T> {
protected final String baseTemporaryFilename;
/**
- * Name separator for temporary files. Temporary files will be named
- * {@code {baseTemporaryFilename}-temp-{bundleId}}.
- */
- protected static final String TEMPORARY_FILENAME_SEPARATOR = "-temp-";
-
- /**
* Build a temporary filename using the temporary filename separator with the given prefix and
* suffix.
*/
protected static final String buildTemporaryFilename(String prefix, String suffix) {
- return prefix + FileBasedWriteOperation.TEMPORARY_FILENAME_SEPARATOR + suffix;
+ try {
+ IOChannelFactory factory = IOChannelUtils.getFactory(prefix);
+ return factory.resolve(prefix, suffix);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
}
/**
@@ -337,7 +339,23 @@ public abstract class FileBasedSink<T> extends Sink<T> {
* @param sink the FileBasedSink that will be used to configure this write operation.
*/
public FileBasedWriteOperation(FileBasedSink<T> sink) {
- this(sink, sink.baseOutputFilename);
+ this(sink, buildTemporaryDirectoryName(sink.getBaseOutputFilename()));
+ }
+
+ private static String buildTemporaryDirectoryName(String baseOutputFilename) {
+ try {
+ IOChannelFactory factory = IOChannelUtils.getFactory(baseOutputFilename);
+ Path baseOutputPath = factory.toPath(baseOutputFilename);
+ return baseOutputPath
+ .resolveSibling(
+ "temp-beam-"
+ + baseOutputPath.getFileName()
+ + "-"
+ + Instant.now().toString(DateTimeFormat.forPattern("yyyy-MM-DD_HH-mm-ss")))
+ .toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
}
/**
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
index a11231b..2d2c0c6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
@@ -148,7 +148,11 @@ public class FileIOChannelFactory implements IOChannelFactory {
@Override
public String resolve(String path, String other) throws IOException {
- Path p = specToFile(path).toPath();
- return p.resolve(other).toString();
+ return toPath(path).resolve(other).toString();
+ }
+
+ @Override
+ public Path toPath(String path) {
+ return specToFile(path).toPath();
}
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
index 14090e3..652e468 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.util;
import java.io.IOException;
import java.nio.channels.ReadableByteChannel;
import java.nio.channels.WritableByteChannel;
+import java.nio.file.Path;
import java.util.Collection;
import java.util.LinkedList;
import java.util.List;
@@ -81,6 +82,11 @@ public class GcsIOChannelFactory implements IOChannelFactory {
@Override
public String resolve(String path, String other) throws IOException {
- return GcsPath.fromUri(path).resolve(other).toString();
+ return toPath(path).resolve(other).toString();
+ }
+
+ @Override
+ public Path toPath(String path) {
+ return GcsPath.fromUri(path);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelFactory.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelFactory.java
index ae6c507..4e55036 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelFactory.java
@@ -21,6 +21,7 @@ import java.io.FileNotFoundException;
import java.io.IOException;
import java.nio.channels.ReadableByteChannel;
import java.nio.channels.WritableByteChannel;
+import java.nio.file.Path;
import java.util.Collection;
/**
@@ -99,4 +100,7 @@ public interface IOChannelFactory {
* dependent and therefore unspecified.
*/
String resolve(String path, String other) throws IOException;
+
+ /** Converts the given string to a {@link Path}. */
+ Path toPath(String path);
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/sdks/java/core/src/main/java/org/apache/beam/sdk/util/gcsfs/GcsPath.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/gcsfs/GcsPath.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/gcsfs/GcsPath.java
index bfcd6da..863b01b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/gcsfs/GcsPath.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/gcsfs/GcsPath.java
@@ -261,7 +261,12 @@ public class GcsPath implements Path {
@Override
public GcsPath getFileName() {
- throw new UnsupportedOperationException();
+ int nameCount = getNameCount();
+ if (nameCount < 2) {
+ throw new UnsupportedOperationException(
+ "Can't get filename from root path in the bucket: " + this);
+ }
+ return getName(nameCount - 1);
}
/**
@@ -436,7 +441,11 @@ public class GcsPath implements Path {
@Override
public Path resolveSibling(String other) {
- throw new UnsupportedOperationException();
+ if (getNameCount() < 2) {
+ throw new UnsupportedOperationException("Can't resolve the sibling of a root path: " + this);
+ }
+ GcsPath parent = getParent();
+ return (parent == null) ? fromUri(other) : parent.resolve(other);
}
@Override
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
index 66bb661..8301afc 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
@@ -89,7 +89,7 @@ public class FileBasedSinkTest {
public void testWriter() throws Exception {
String testUid = "testId";
String expectedFilename =
- getBaseTempFilename() + FileBasedWriteOperation.TEMPORARY_FILENAME_SEPARATOR + testUid;
+ getBaseTempFilename() + "/" + testUid;
SimpleSink.SimpleWriter writer = buildWriter();
List<String> values = Arrays.asList("sympathetic vulture", "boresome hummingbird");
@@ -193,8 +193,7 @@ public class FileBasedSinkTest {
runFinalize(writeOp, files, false);
// create a temporary file
- tmpFolder.newFile(
- baseTemporaryFilename + FileBasedWriteOperation.TEMPORARY_FILENAME_SEPARATOR + "1");
+ tmpFolder.newFile(baseTemporaryFilename + "/1");
runFinalize(writeOp, files, false);
}
@@ -217,7 +216,9 @@ public class FileBasedSinkTest {
for (int i = 0; i < numFiles; i++) {
String temporaryFilename =
FileBasedWriteOperation.buildTemporaryFilename(baseTemporaryFilename, "" + i);
- File tmpFile = tmpFolder.newFile(temporaryFilename);
+ File tmpFile = new File(tmpFolder.getRoot(), temporaryFilename);
+ tmpFile.getParentFile().mkdirs();
+ assertTrue(tmpFile.createNewFile());
temporaryFiles.add(tmpFile);
}
@@ -263,8 +264,10 @@ public class FileBasedSinkTest {
List<File> temporaryFiles = new ArrayList<>();
List<File> outputFiles = new ArrayList<>();
for (int i = 0; i < numFiles; i++) {
- File tmpFile = tmpFolder.newFile(
+ File tmpFile = new File(tmpFolder.getRoot(),
FileBasedWriteOperation.buildTemporaryFilename(baseTemporaryFilename, "" + i));
+ tmpFile.getParentFile().mkdirs();
+ assertTrue(tmpFile.createNewFile());
temporaryFiles.add(tmpFile);
File outputFile = tmpFolder.newFile(baseOutputFilename + i);
outputFiles.add(outputFile);
@@ -496,11 +499,13 @@ public class FileBasedSinkTest {
@Test
public void testFileBasedWriterWithWritableByteChannelFactory() throws Exception {
final String testUid = "testId";
- final String expectedFilename =
- getBaseOutputFilename() + FileBasedWriteOperation.TEMPORARY_FILENAME_SEPARATOR + testUid;
- final FileBasedWriter<String> writer =
+ SimpleSink.SimpleWriteOperation writeOp =
new SimpleSink(getBaseOutputFilename(), "txt", new DrunkWritableByteChannelFactory())
- .createWriteOperation(null).createWriter(null);
+ .createWriteOperation(null);
+ final FileBasedWriter<String> writer =
+ writeOp.createWriter(null);
+ final String expectedFilename =
+ writeOp.baseTemporaryFilename + "/" + testUid;
final List<String> expected = new ArrayList<>();
expected.add("header");
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/sdks/java/core/src/test/java/org/apache/beam/sdk/io/XmlSinkTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/XmlSinkTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/XmlSinkTest.java
index 2788ea6..653a9d0 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/XmlSinkTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/XmlSinkTest.java
@@ -19,6 +19,7 @@ package org.apache.beam.sdk.io;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsString;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
@@ -28,6 +29,7 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.nio.channels.WritableByteChannel;
+import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
@@ -59,7 +61,7 @@ public class XmlSinkTest {
private Class<Bird> testClass = Bird.class;
private String testRootElement = "testElement";
- private String testFilePrefix = "testPrefix";
+ private String testFilePrefix = "/path/to/testPrefix";
/**
* An XmlWriter correctly writes objects as Xml elements with an enclosing root element.
@@ -143,7 +145,11 @@ public class XmlSinkTest {
assertEquals(testFilePrefix, writeOp.getSink().baseOutputFilename);
assertEquals(testRootElement, writeOp.getSink().rootElementName);
assertEquals(XmlSink.XML_EXTENSION, writeOp.getSink().extension);
- assertEquals(testFilePrefix, writeOp.baseTemporaryFilename);
+ Path outputPath = new File(testFilePrefix).toPath();
+ Path tempPath = new File(writeOp.baseTemporaryFilename).toPath();
+ assertEquals(outputPath.getParent(), tempPath.getParent());
+ assertThat(
+ tempPath.getFileName().toString(), containsString("temp-beam-" + outputPath.getFileName()));
}
/**
@@ -156,7 +162,11 @@ public class XmlSinkTest {
XmlSink.writeOf(testClass, testRootElement, testFilePrefix)
.createWriteOperation(options);
XmlWriter<Bird> writer = writeOp.createWriter(options);
- assertEquals(testFilePrefix, writer.getWriteOperation().baseTemporaryFilename);
+ Path outputPath = new File(testFilePrefix).toPath();
+ Path tempPath = new File(writer.getWriteOperation().baseTemporaryFilename).toPath();
+ assertEquals(outputPath.getParent(), tempPath.getParent());
+ assertThat(
+ tempPath.getFileName().toString(), containsString("temp-beam-" + outputPath.getFileName()));
assertEquals(testRootElement, writer.getWriteOperation().getSink().rootElementName);
assertNotNull(writer.marshaller);
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c1008c4/sdks/java/core/src/test/java/org/apache/beam/sdk/util/gcsfs/GcsPathTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/gcsfs/GcsPathTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/gcsfs/GcsPathTest.java
index 5c86184..426fb16 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/gcsfs/GcsPathTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/gcsfs/GcsPathTest.java
@@ -31,7 +31,9 @@ import java.util.Iterator;
import java.util.List;
import org.hamcrest.Matchers;
import org.junit.Assert;
+import org.junit.Rule;
import org.junit.Test;
+import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -73,6 +75,9 @@ public class GcsPathTest {
new TestCase("gs://bucket/", "bucket")
);
+ @Rule
+ public ExpectedException thrown = ExpectedException.none();
+
@Test
public void testGcsPathParsing() throws Exception {
for (TestCase testCase : PATH_TEST_CASES) {
@@ -237,6 +242,26 @@ public class GcsPathTest {
}
@Test
+ public void testGetFileName() {
+ assertEquals("foo", GcsPath.fromUri("gs://bucket/bar/foo").getFileName().toString());
+ assertEquals("foo", GcsPath.fromUri("gs://bucket/foo").getFileName().toString());
+ thrown.expect(UnsupportedOperationException.class);
+ GcsPath.fromUri("gs://bucket/").getFileName();
+ }
+
+ @Test
+ public void testResolveSibling() {
+ assertEquals(
+ "gs://bucket/bar/moo",
+ GcsPath.fromUri("gs://bucket/bar/foo").resolveSibling("moo").toString());
+ assertEquals(
+ "gs://bucket/moo",
+ GcsPath.fromUri("gs://bucket/foo").resolveSibling("moo").toString());
+ thrown.expect(UnsupportedOperationException.class);
+ GcsPath.fromUri("gs://bucket/").resolveSibling("moo");
+ }
+
+ @Test
public void testCompareTo() {
GcsPath a = GcsPath.fromComponents("bucket", "a");
GcsPath b = GcsPath.fromComponents("bucket", "b");
[20/50] incubator-beam git commit: Implement
GroupAlsoByWindowsEvaluator Directly
Posted by th...@apache.org.
Implement GroupAlsoByWindowsEvaluator Directly
Invoke the ReduceFnRunner methods directly rather than by constructing a
GroupAlsoByWindowDoFn and a ParDoEvaluator. Instead, construct a
ReduceFnRunner, ReduceFn, and invoke the methods directly.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6b36feab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6b36feab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6b36feab
Branch: refs/heads/apex-runner
Commit: 6b36feabd081ee7e174354e37b881abe8b38574e
Parents: 2a1fdee
Author: Thomas Groh <tg...@google.com>
Authored: Tue Nov 1 14:29:48 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Thu Nov 3 15:15:58 2016 -0700
----------------------------------------------------------------------
.../runners/direct/AggregatorContainer.java | 20 +-
.../GroupAlsoByWindowEvaluatorFactory.java | 248 +++++++++++++++----
2 files changed, 218 insertions(+), 50 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b36feab/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AggregatorContainer.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AggregatorContainer.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AggregatorContainer.java
index 03e2509..7b6bc64 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AggregatorContainer.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AggregatorContainer.java
@@ -151,8 +151,24 @@ public class AggregatorContainer {
@Override
public <InputT, AccumT, OutputT> Aggregator<InputT, OutputT> createAggregatorForDoFn(
- Class<?> fnClass, ExecutionContext.StepContext step,
- String name, CombineFn<InputT, AccumT, OutputT> combine) {
+ Class<?> fnClass,
+ ExecutionContext.StepContext step,
+ String name,
+ CombineFn<InputT, AccumT, OutputT> combine) {
+ return createAggregatorForStep(step, name, combine);
+ }
+
+ public <InputT, AccumT, OutputT> Aggregator<InputT, OutputT> createSystemAggregator(
+ ExecutionContext.StepContext step,
+ String name,
+ CombineFn<InputT, AccumT, OutputT> combiner) {
+ return createAggregatorForStep(step, name, combiner);
+ }
+
+ private <InputT, AccumT, OutputT> Aggregator<InputT, OutputT> createAggregatorForStep(
+ ExecutionContext.StepContext step,
+ String name,
+ CombineFn<InputT, AccumT, OutputT> combine) {
checkState(!committed, "Cannot create aggregators after committing");
AggregatorKey key = AggregatorKey.create(step.getStepName(), name);
AggregatorInfo<?, ?, ?> aggregatorInfo = accumulatorDeltas.get(key);
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b36feab/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
index 37cc319..e5c5e4b 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
@@ -17,30 +17,47 @@
*/
package org.apache.beam.runners.direct;
-import com.google.common.collect.ImmutableMap;
-import java.util.Collections;
-import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetDoFn;
+import com.google.common.base.Function;
+import com.google.common.base.Predicate;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import org.apache.beam.runners.core.GroupAlsoByWindowsDoFn;
import org.apache.beam.runners.core.GroupByKeyViaGroupByKeyOnly;
import org.apache.beam.runners.core.GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow;
import org.apache.beam.runners.core.GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly;
+import org.apache.beam.runners.core.ReduceFnRunner;
import org.apache.beam.runners.core.SystemReduceFn;
+import org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachine;
+import org.apache.beam.runners.core.triggers.TriggerStateMachines;
import org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext;
import org.apache.beam.runners.direct.DirectGroupByKey.DirectGroupAlsoByWindow;
import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
+import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.transforms.Aggregator;
import org.apache.beam.sdk.transforms.AppliedPTransform;
-import org.apache.beam.sdk.transforms.OldDoFn;
import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.KeyedWorkItem;
+import org.apache.beam.sdk.util.TimerInternals;
+import org.apache.beam.sdk.util.TimerInternals.TimerData;
+import org.apache.beam.sdk.util.WindowTracing;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowingInternals;
import org.apache.beam.sdk.util.WindowingStrategy;
+import org.apache.beam.sdk.util.state.CopyOnAccessInMemoryStateInternals;
import org.apache.beam.sdk.util.state.StateInternals;
-import org.apache.beam.sdk.util.state.StateInternalsFactory;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
+import org.joda.time.Instant;
/**
* The {@link DirectRunner} {@link TransformEvaluatorFactory} for the
@@ -86,9 +103,23 @@ class GroupAlsoByWindowEvaluatorFactory implements TransformEvaluatorFactory {
*/
private static class GroupAlsoByWindowEvaluator<K, V>
implements TransformEvaluator<KeyedWorkItem<K, V>> {
- private static final TupleTag<Object> MAIN_OUTPUT_TAG = new TupleTag<Object>() {};
+ private final EvaluationContext evaluationContext;
+ private final AppliedPTransform<
+ PCollection<KeyedWorkItem<K, V>>, PCollection<KV<K, Iterable<V>>>,
+ DirectGroupAlsoByWindow<K, V>>
+ application;
- private final TransformEvaluator<KeyedWorkItem<K, V>> gabwParDoEvaluator;
+ private final DirectStepContext stepContext;
+ private @SuppressWarnings("unchecked") final WindowingStrategy<?, BoundedWindow>
+ windowingStrategy;
+
+ private final Collection<UncommittedBundle<?>> outputBundles;
+ private final ImmutableList.Builder<WindowedValue<KeyedWorkItem<K, V>>> unprocessedElements;
+ private final AggregatorContainer.Mutator aggregatorChanges;
+
+ private final SystemReduceFn<K, V, Iterable<V>, Iterable<V>, BoundedWindow> reduceFn;
+ private final Aggregator<Long, Long> droppedDueToClosedWindow;
+ private final Aggregator<Long, Long> droppedDueToLateness;
public GroupAlsoByWindowEvaluator(
final EvaluationContext evaluationContext,
@@ -97,67 +128,188 @@ class GroupAlsoByWindowEvaluatorFactory implements TransformEvaluatorFactory {
PCollection<KeyedWorkItem<K, V>>,
PCollection<KV<K, Iterable<V>>>,
DirectGroupAlsoByWindow<K, V>> application) {
+ this.evaluationContext = evaluationContext;
+ this.application = application;
- Coder<V> valueCoder =
- application.getTransform().getValueCoder(inputBundle.getPCollection().getCoder());
-
- @SuppressWarnings("unchecked")
- WindowingStrategy<?, BoundedWindow> windowingStrategy =
+ stepContext = evaluationContext
+ .getExecutionContext(application, inputBundle.getKey())
+ .getOrCreateStepContext(
+ evaluationContext.getStepName(application), application.getTransform().getName());
+ windowingStrategy =
(WindowingStrategy<?, BoundedWindow>)
application.getTransform().getInputWindowingStrategy();
- DirectStepContext stepContext =
- evaluationContext
- .getExecutionContext(application, inputBundle.getKey())
- .getOrCreateStepContext(
- evaluationContext.getStepName(application), application.getTransform().getName());
+ outputBundles = new ArrayList<>();
+ unprocessedElements = ImmutableList.builder();
+ aggregatorChanges = evaluationContext.getAggregatorMutator();
- StateInternals<K> stateInternals = (StateInternals<K>) stepContext.stateInternals();
-
- OldDoFn<KeyedWorkItem<K, V>, KV<K, Iterable<V>>> gabwDoFn =
- GroupAlsoByWindowViaWindowSetDoFn.create(
- windowingStrategy,
- new ConstantStateInternalsFactory<K>(stateInternals),
- SystemReduceFn.<K, V, BoundedWindow>buffering(valueCoder));
-
- // Not technically legit, as the application is not a ParDo
- this.gabwParDoEvaluator =
- ParDoEvaluator.create(
- evaluationContext,
- stepContext,
- application,
- windowingStrategy,
- gabwDoFn,
- Collections.<PCollectionView<?>>emptyList(),
- MAIN_OUTPUT_TAG,
- Collections.<TupleTag<?>>emptyList(),
- ImmutableMap.<TupleTag<?>, PCollection<?>>of(
- MAIN_OUTPUT_TAG, application.getOutput()));
+ Coder<V> valueCoder =
+ application.getTransform().getValueCoder(inputBundle.getPCollection().getCoder());
+ reduceFn = SystemReduceFn.buffering(valueCoder);
+ droppedDueToClosedWindow = aggregatorChanges.createSystemAggregator(stepContext,
+ GroupAlsoByWindowsDoFn.DROPPED_DUE_TO_CLOSED_WINDOW_COUNTER,
+ new Sum.SumLongFn());
+ droppedDueToLateness = aggregatorChanges.createSystemAggregator(stepContext,
+ GroupAlsoByWindowsDoFn.DROPPED_DUE_TO_LATENESS_COUNTER,
+ new Sum.SumLongFn());
}
@Override
public void processElement(WindowedValue<KeyedWorkItem<K, V>> element) throws Exception {
- gabwParDoEvaluator.processElement(element);
+ KeyedWorkItem<K, V> workItem = element.getValue();
+ K key = workItem.key();
+
+ UncommittedBundle<KV<K, Iterable<V>>> bundle =
+ evaluationContext.createBundle(application.getOutput());
+ outputBundles.add(bundle);
+ CopyOnAccessInMemoryStateInternals<K> stateInternals =
+ (CopyOnAccessInMemoryStateInternals<K>) stepContext.stateInternals();
+ DirectTimerInternals timerInternals = stepContext.timerInternals();
+ ReduceFnRunner<K, V, Iterable<V>, BoundedWindow> reduceFnRunner =
+ new ReduceFnRunner<>(
+ key,
+ windowingStrategy,
+ ExecutableTriggerStateMachine.create(
+ TriggerStateMachines.stateMachineForTrigger(windowingStrategy.getTrigger())),
+ stateInternals,
+ timerInternals,
+ new DirectWindowingInternals<>(bundle),
+ droppedDueToClosedWindow,
+ reduceFn,
+ evaluationContext.getPipelineOptions());
+
+ // Drop any elements within expired windows
+ reduceFnRunner.processElements(
+ dropExpiredWindows(key, workItem.elementsIterable(), timerInternals));
+ for (TimerData timer : workItem.timersIterable()) {
+ reduceFnRunner.onTimer(timer);
+ }
+ reduceFnRunner.persist();
}
@Override
public TransformResult finishBundle() throws Exception {
- return gabwParDoEvaluator.finishBundle();
+ // State is initialized within the constructor. It can never be null.
+ CopyOnAccessInMemoryStateInternals<?> state = stepContext.commitState();
+ return StepTransformResult.withHold(application, state.getEarliestWatermarkHold())
+ .withState(state)
+ .addOutput(outputBundles)
+ .withTimerUpdate(stepContext.getTimerUpdate())
+ .withAggregatorChanges(aggregatorChanges)
+ .addUnprocessedElements(unprocessedElements.build())
+ .build();
+ }
+
+ /**
+ * Returns an {@code Iterable<WindowedValue<InputT>>} that only contains non-late input
+ * elements.
+ */
+ public Iterable<WindowedValue<V>> dropExpiredWindows(
+ final K key, Iterable<WindowedValue<V>> elements, final TimerInternals timerInternals) {
+ return FluentIterable.from(elements)
+ .transformAndConcat(
+ // Explode windows to filter out expired ones
+ new Function<WindowedValue<V>, Iterable<WindowedValue<V>>>() {
+ @Override
+ public Iterable<WindowedValue<V>> apply(WindowedValue<V> input) {
+ return input.explodeWindows();
+ }
+ })
+ .filter(
+ new Predicate<WindowedValue<V>>() {
+ @Override
+ public boolean apply(WindowedValue<V> input) {
+ BoundedWindow window = Iterables.getOnlyElement(input.getWindows());
+ boolean expired =
+ window
+ .maxTimestamp()
+ .plus(windowingStrategy.getAllowedLateness())
+ .isBefore(timerInternals.currentInputWatermarkTime());
+ if (expired) {
+ // The element is too late for this window.
+ droppedDueToLateness.addValue(1L);
+ WindowTracing.debug(
+ "GroupAlsoByWindow: Dropping element at {} for key: {}; "
+ + "window: {} since it is too far behind inputWatermark: {}",
+ input.getTimestamp(),
+ key,
+ window,
+ timerInternals.currentInputWatermarkTime());
+ }
+ // Keep the element if the window is not expired.
+ return !expired;
+ }
+ });
}
}
- private static final class ConstantStateInternalsFactory<K>
- implements StateInternalsFactory<K> {
- private final StateInternals<K> stateInternals;
+ private static class DirectWindowingInternals<K, V>
+ implements WindowingInternals<Object, KV<K, Iterable<V>>> {
+ private final UncommittedBundle<KV<K, Iterable<V>>> bundle;
+
+ private DirectWindowingInternals(
+ UncommittedBundle<KV<K, Iterable<V>>> bundle) {
+ this.bundle = bundle;
+ }
- private ConstantStateInternalsFactory(StateInternals<K> stateInternals) {
- this.stateInternals = stateInternals;
+ @Override
+ public StateInternals<?> stateInternals() {
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s should use the %s it is provided rather than the contents of %s",
+ ReduceFnRunner.class.getSimpleName(),
+ StateInternals.class.getSimpleName(),
+ getClass().getSimpleName()));
+ }
+
+ @Override
+ public void outputWindowedValue(
+ KV<K, Iterable<V>> output,
+ Instant timestamp,
+ Collection<? extends BoundedWindow> windows,
+ PaneInfo pane) {
+ bundle.add(WindowedValue.of(output, timestamp, windows, pane));
+ }
+
+ @Override
+ public TimerInternals timerInternals() {
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s should use the %s it is provided rather than the contents of %s",
+ ReduceFnRunner.class.getSimpleName(),
+ TimerInternals.class.getSimpleName(),
+ getClass().getSimpleName()));
+ }
+
+ @Override
+ public Collection<? extends BoundedWindow> windows() {
+ throw new IllegalArgumentException(
+ String.format(
+ "%s should not access Windows via %s.windows(); "
+ + "it should instead inspect the window of the input elements",
+ GroupAlsoByWindowEvaluator.class.getSimpleName(),
+ WindowingInternals.class.getSimpleName()));
+ }
+
+ @Override
+ public PaneInfo pane() {
+ throw new IllegalArgumentException(
+ String.format(
+ "%s should not access Windows via %s.windows(); "
+ + "it should instead inspect the window of the input elements",
+ GroupAlsoByWindowEvaluator.class.getSimpleName(),
+ WindowingInternals.class.getSimpleName()));
+ }
+
+ @Override
+ public <T> void writePCollectionViewData(
+ TupleTag<?> tag, Iterable<WindowedValue<T>> data, Coder<T> elemCoder) throws IOException {
+ throw new UnsupportedOperationException();
}
@Override
- @SuppressWarnings("unchecked")
- public StateInternals<K> stateInternalsForKey(K key) {
- return stateInternals;
+ public <T> T sideInput(PCollectionView<T> view, BoundedWindow mainInputWindow) {
+ throw new UnsupportedOperationException();
}
}
}
[30/50] incubator-beam git commit: Closes #1285
Posted by th...@apache.org.
Closes #1285
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f06deac8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f06deac8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f06deac8
Branch: refs/heads/apex-runner
Commit: f06deac8c10f10b4b64ddc51358b1b551845e595
Parents: faf55c7 f50b214
Author: Dan Halperin <dh...@google.com>
Authored: Fri Nov 4 10:51:29 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Fri Nov 4 10:51:29 2016 -0700
----------------------------------------------------------------------
examples/java/pom.xml | 5 -----
examples/java8/pom.xml | 5 -----
examples/pom.xml | 13 ++++++++++++-
pom.xml | 3 +++
runners/core-java/pom.xml | 5 -----
runners/direct-java/pom.xml | 5 -----
runners/flink/examples/pom.xml | 5 -----
runners/flink/runner/pom.xml | 5 -----
runners/google-cloud-dataflow-java/pom.xml | 7 -------
runners/pom.xml | 11 +++++++++++
runners/spark/pom.xml | 4 ----
sdks/java/build-tools/pom.xml | 4 ++--
.../src/main/resources/beam/checkstyle.xml | 2 +-
sdks/java/core/pom.xml | 18 +++++++++---------
sdks/java/extensions/join-library/pom.xml | 4 ----
sdks/java/extensions/sorter/pom.xml | 4 ----
sdks/java/io/google-cloud-platform/pom.xml | 4 ----
sdks/java/io/hdfs/pom.xml | 4 ----
sdks/java/io/jdbc/pom.xml | 6 +-----
sdks/java/io/jms/pom.xml | 4 ----
sdks/java/io/kafka/pom.xml | 4 ----
sdks/java/io/kinesis/pom.xml | 4 ----
sdks/java/io/mongodb/pom.xml | 4 ----
sdks/java/java8tests/pom.xml | 5 -----
sdks/java/microbenchmarks/pom.xml | 5 -----
sdks/java/pom.xml | 4 +++-
sdks/pom.xml | 16 +++++++++++++++-
27 files changed, 57 insertions(+), 103 deletions(-)
----------------------------------------------------------------------
[43/50] incubator-beam git commit: Closes #1292
Posted by th...@apache.org.
Closes #1292
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/70255d26
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/70255d26
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/70255d26
Branch: refs/heads/apex-runner
Commit: 70255d26880ce6443b8a931a6a778040fdbc5339
Parents: 5864a38 68f2638
Author: Dan Halperin <dh...@google.com>
Authored: Mon Nov 7 08:51:40 2016 -0800
Committer: Dan Halperin <dh...@google.com>
Committed: Mon Nov 7 08:51:40 2016 -0800
----------------------------------------------------------------------
examples/java/pom.xml | 6 ------
examples/java8/pom.xml | 6 ------
pom.xml | 5 +++++
runners/core-java/pom.xml | 6 ------
runners/direct-java/pom.xml | 6 ------
runners/flink/examples/pom.xml | 4 ----
runners/flink/runner/pom.xml | 4 ----
runners/google-cloud-dataflow-java/pom.xml | 6 ------
runners/spark/pom.xml | 4 ----
sdks/java/core/pom.xml | 6 ------
sdks/java/extensions/join-library/pom.xml | 4 ----
sdks/java/extensions/sorter/pom.xml | 4 ----
sdks/java/io/google-cloud-platform/pom.xml | 4 ----
sdks/java/io/hdfs/pom.xml | 4 ----
sdks/java/io/jdbc/pom.xml | 4 ----
sdks/java/io/jms/pom.xml | 4 ----
sdks/java/io/kafka/pom.xml | 4 ----
sdks/java/io/kinesis/pom.xml | 4 ----
sdks/java/io/mongodb/pom.xml | 4 ----
sdks/java/java8tests/pom.xml | 6 ------
sdks/java/microbenchmarks/pom.xml | 5 -----
21 files changed, 5 insertions(+), 95 deletions(-)
----------------------------------------------------------------------
[32/50] incubator-beam git commit: Closes #1284
Posted by th...@apache.org.
Closes #1284
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6e1e57b0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6e1e57b0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6e1e57b0
Branch: refs/heads/apex-runner
Commit: 6e1e57b09f4fbab021eb0ebb0748accc8bd90d0a
Parents: f06deac f19a25d
Author: Dan Halperin <dh...@google.com>
Authored: Fri Nov 4 10:56:40 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Fri Nov 4 10:56:40 2016 -0700
----------------------------------------------------------------------
pom.xml | 4 ----
sdks/java/pom.xml | 5 +----
2 files changed, 1 insertion(+), 8 deletions(-)
----------------------------------------------------------------------
[27/50] incubator-beam git commit: Switch DoFnSignature, etc,
from TypeToken to TypeDescriptor
Posted by th...@apache.org.
Switch DoFnSignature, etc, from TypeToken to TypeDescriptor
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8336b24c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8336b24c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8336b24c
Branch: refs/heads/apex-runner
Commit: 8336b24c97c620fa3edb02301299080bda96379a
Parents: d936ed8
Author: Kenneth Knowles <kl...@google.com>
Authored: Tue Nov 1 14:48:54 2016 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Thu Nov 3 21:32:53 2016 -0700
----------------------------------------------------------------------
.../sdk/transforms/reflect/DoFnInvokers.java | 7 +-
.../sdk/transforms/reflect/DoFnSignature.java | 23 ++-
.../sdk/transforms/reflect/DoFnSignatures.java | 177 ++++++++++---------
.../DoFnSignaturesSplittableDoFnTest.java | 18 +-
.../transforms/reflect/DoFnSignaturesTest.java | 7 +-
.../reflect/DoFnSignaturesTestUtils.java | 8 +-
6 files changed, 124 insertions(+), 116 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8336b24c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
index dd134b7..c5a23dc 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
@@ -19,7 +19,6 @@ package org.apache.beam.sdk.transforms.reflect;
import static com.google.common.base.Preconditions.checkArgument;
-import com.google.common.reflect.TypeToken;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
@@ -263,9 +262,9 @@ public class DoFnInvokers {
/** Default implementation of {@link DoFn.GetRestrictionCoder}, for delegation by bytebuddy. */
public static class DefaultRestrictionCoder {
- private final TypeToken<?> restrictionType;
+ private final TypeDescriptor<?> restrictionType;
- DefaultRestrictionCoder(TypeToken<?> restrictionType) {
+ DefaultRestrictionCoder(TypeDescriptor<?> restrictionType) {
this.restrictionType = restrictionType;
}
@@ -273,7 +272,7 @@ public class DoFnInvokers {
@SuppressWarnings({"unused", "unchecked"})
public <RestrictionT> Coder<RestrictionT> invokeGetRestrictionCoder(CoderRegistry registry)
throws CannotProvideCoderException {
- return (Coder) registry.getCoder(TypeDescriptor.of(restrictionType.getType()));
+ return (Coder) registry.getCoder(restrictionType);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8336b24c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index 71f7e53..6b98805 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -20,7 +20,6 @@ package org.apache.beam.sdk.transforms.reflect;
import com.google.auto.value.AutoValue;
import com.google.common.base.Predicates;
import com.google.common.collect.Iterables;
-import com.google.common.reflect.TypeToken;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.Collections;
@@ -342,7 +341,7 @@ public abstract class DoFnSignature {
/** Concrete type of the {@link RestrictionTracker} parameter, if present. */
@Nullable
- abstract TypeToken<?> trackerT();
+ abstract TypeDescriptor<?> trackerT();
/** Whether this {@link DoFn} returns a {@link ProcessContinuation} or void. */
public abstract boolean hasReturnValue();
@@ -350,7 +349,7 @@ public abstract class DoFnSignature {
static ProcessElementMethod create(
Method targetMethod,
List<Parameter> extraParameters,
- TypeToken<?> trackerT,
+ TypeDescriptor<?> trackerT,
boolean hasReturnValue) {
return new AutoValue_DoFnSignature_ProcessElementMethod(
targetMethod, Collections.unmodifiableList(extraParameters), trackerT, hasReturnValue);
@@ -462,9 +461,9 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Type of the returned restriction. */
- abstract TypeToken<?> restrictionT();
+ abstract TypeDescriptor<?> restrictionT();
- static GetInitialRestrictionMethod create(Method targetMethod, TypeToken<?> restrictionT) {
+ static GetInitialRestrictionMethod create(Method targetMethod, TypeDescriptor<?> restrictionT) {
return new AutoValue_DoFnSignature_GetInitialRestrictionMethod(targetMethod, restrictionT);
}
}
@@ -477,9 +476,9 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Type of the restriction taken and returned. */
- abstract TypeToken<?> restrictionT();
+ abstract TypeDescriptor<?> restrictionT();
- static SplitRestrictionMethod create(Method targetMethod, TypeToken<?> restrictionT) {
+ static SplitRestrictionMethod create(Method targetMethod, TypeDescriptor<?> restrictionT) {
return new AutoValue_DoFnSignature_SplitRestrictionMethod(targetMethod, restrictionT);
}
}
@@ -492,13 +491,13 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Type of the input restriction. */
- abstract TypeToken<?> restrictionT();
+ abstract TypeDescriptor<?> restrictionT();
/** Type of the returned {@link RestrictionTracker}. */
- abstract TypeToken<?> trackerT();
+ abstract TypeDescriptor<?> trackerT();
static NewTrackerMethod create(
- Method targetMethod, TypeToken<?> restrictionT, TypeToken<?> trackerT) {
+ Method targetMethod, TypeDescriptor<?> restrictionT, TypeDescriptor<?> trackerT) {
return new AutoValue_DoFnSignature_NewTrackerMethod(targetMethod, restrictionT, trackerT);
}
}
@@ -511,9 +510,9 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Type of the returned {@link Coder}. */
- abstract TypeToken<?> coderT();
+ abstract TypeDescriptor<?> coderT();
- static GetRestrictionCoderMethod create(Method targetMethod, TypeToken<?> coderT) {
+ static GetRestrictionCoderMethod create(Method targetMethod, TypeDescriptor<?> coderT) {
return new AutoValue_DoFnSignature_GetRestrictionCoderMethod(targetMethod, coderT);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8336b24c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
index 5814c0e..c690ace 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
@@ -22,8 +22,6 @@ import static com.google.common.base.Preconditions.checkState;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Maps;
-import com.google.common.reflect.TypeParameter;
-import com.google.common.reflect.TypeToken;
import java.lang.annotation.Annotation;
import java.lang.reflect.AnnotatedElement;
import java.lang.reflect.Field;
@@ -57,6 +55,7 @@ import org.apache.beam.sdk.util.state.State;
import org.apache.beam.sdk.util.state.StateSpec;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeParameter;
/**
* Parses a {@link DoFn} and computes its {@link DoFnSignature}. See {@link #getSignature}.
@@ -90,18 +89,18 @@ public class DoFnSignatures {
errors.checkArgument(DoFn.class.isAssignableFrom(fnClass), "Must be subtype of DoFn");
builder.setFnClass(fnClass);
- TypeToken<? extends DoFn<?, ?>> fnToken = TypeToken.of(fnClass);
+ TypeDescriptor<? extends DoFn<?, ?>> fnT = TypeDescriptor.of(fnClass);
// Extract the input and output type, and whether the fn is bounded.
- TypeToken<?> inputT = null;
- TypeToken<?> outputT = null;
- for (TypeToken<?> supertype : fnToken.getTypes()) {
+ TypeDescriptor<?> inputT = null;
+ TypeDescriptor<?> outputT = null;
+ for (TypeDescriptor<?> supertype : fnT.getTypes()) {
if (!supertype.getRawType().equals(DoFn.class)) {
continue;
}
Type[] args = ((ParameterizedType) supertype.getType()).getActualTypeArguments();
- inputT = TypeToken.of(args[0]);
- outputT = TypeToken.of(args[1]);
+ inputT = TypeDescriptor.of(args[0]);
+ outputT = TypeDescriptor.of(args[1]);
}
errors.checkNotNull(inputT, "Unable to determine input type");
@@ -169,7 +168,7 @@ public class DoFnSignatures {
DoFnSignature.ProcessElementMethod processElement =
analyzeProcessElementMethod(
processElementErrors,
- fnToken,
+ fnT,
processElementMethod,
inputT,
outputT,
@@ -180,14 +179,14 @@ public class DoFnSignatures {
if (startBundleMethod != null) {
ErrorReporter startBundleErrors = errors.forMethod(DoFn.StartBundle.class, startBundleMethod);
builder.setStartBundle(
- analyzeBundleMethod(startBundleErrors, fnToken, startBundleMethod, inputT, outputT));
+ analyzeBundleMethod(startBundleErrors, fnT, startBundleMethod, inputT, outputT));
}
if (finishBundleMethod != null) {
ErrorReporter finishBundleErrors =
errors.forMethod(DoFn.FinishBundle.class, finishBundleMethod);
builder.setFinishBundle(
- analyzeBundleMethod(finishBundleErrors, fnToken, finishBundleMethod, inputT, outputT));
+ analyzeBundleMethod(finishBundleErrors, fnT, finishBundleMethod, inputT, outputT));
}
if (setupMethod != null) {
@@ -209,7 +208,7 @@ public class DoFnSignatures {
builder.setGetInitialRestriction(
getInitialRestriction =
analyzeGetInitialRestrictionMethod(
- getInitialRestrictionErrors, fnToken, getInitialRestrictionMethod, inputT));
+ getInitialRestrictionErrors, fnT, getInitialRestrictionMethod, inputT));
}
DoFnSignature.SplitRestrictionMethod splitRestriction = null;
@@ -219,7 +218,7 @@ public class DoFnSignatures {
builder.setSplitRestriction(
splitRestriction =
analyzeSplitRestrictionMethod(
- splitRestrictionErrors, fnToken, splitRestrictionMethod, inputT));
+ splitRestrictionErrors, fnT, splitRestrictionMethod, inputT));
}
DoFnSignature.GetRestrictionCoderMethod getRestrictionCoder = null;
@@ -229,17 +228,17 @@ public class DoFnSignatures {
builder.setGetRestrictionCoder(
getRestrictionCoder =
analyzeGetRestrictionCoderMethod(
- getRestrictionCoderErrors, fnToken, getRestrictionCoderMethod));
+ getRestrictionCoderErrors, fnT, getRestrictionCoderMethod));
}
DoFnSignature.NewTrackerMethod newTracker = null;
if (newTrackerMethod != null) {
ErrorReporter newTrackerErrors = errors.forMethod(DoFn.NewTracker.class, newTrackerMethod);
builder.setNewTracker(
- newTracker = analyzeNewTrackerMethod(newTrackerErrors, fnToken, newTrackerMethod));
+ newTracker = analyzeNewTrackerMethod(newTrackerErrors, fnT, newTrackerMethod));
}
- builder.setIsBoundedPerElement(inferBoundedness(fnToken, processElement, errors));
+ builder.setIsBoundedPerElement(inferBoundedness(fnT, processElement, errors));
DoFnSignature signature = builder.build();
@@ -271,11 +270,11 @@ public class DoFnSignatures {
* </ol>
*/
private static PCollection.IsBounded inferBoundedness(
- TypeToken<? extends DoFn> fnToken,
+ TypeDescriptor<? extends DoFn> fnT,
DoFnSignature.ProcessElementMethod processElement,
ErrorReporter errors) {
PCollection.IsBounded isBounded = null;
- for (TypeToken<?> supertype : fnToken.getTypes()) {
+ for (TypeDescriptor<?> supertype : fnT.getTypes()) {
if (supertype.getRawType().isAnnotationPresent(DoFn.BoundedPerElement.class)
|| supertype.getRawType().isAnnotationPresent(DoFn.UnboundedPerElement.class)) {
errors.checkArgument(
@@ -354,7 +353,7 @@ public class DoFnSignatures {
ErrorReporter getInitialRestrictionErrors =
errors.forMethod(DoFn.GetInitialRestriction.class, getInitialRestriction.targetMethod());
- TypeToken<?> restrictionT = getInitialRestriction.restrictionT();
+ TypeDescriptor<?> restrictionT = getInitialRestriction.restrictionT();
getInitialRestrictionErrors.checkArgument(
restrictionT.equals(newTracker.restrictionT()),
@@ -411,49 +410,54 @@ public class DoFnSignatures {
}
/**
- * Generates a type token for {@code DoFn<InputT, OutputT>.ProcessContext} given {@code InputT}
- * and {@code OutputT}.
+ * Generates a {@link TypeDescriptor} for {@code DoFn<InputT, OutputT>.ProcessContext} given
+ * {@code InputT} and {@code OutputT}.
*/
private static <InputT, OutputT>
- TypeToken<DoFn<InputT, OutputT>.ProcessContext> doFnProcessContextTypeOf(
- TypeToken<InputT> inputT, TypeToken<OutputT> outputT) {
- return new TypeToken<DoFn<InputT, OutputT>.ProcessContext>() {}.where(
+ TypeDescriptor<DoFn<InputT, OutputT>.ProcessContext> doFnProcessContextTypeOf(
+ TypeDescriptor<InputT> inputT, TypeDescriptor<OutputT> outputT) {
+ return new TypeDescriptor<DoFn<InputT, OutputT>.ProcessContext>() {}.where(
new TypeParameter<InputT>() {}, inputT)
.where(new TypeParameter<OutputT>() {}, outputT);
}
/**
- * Generates a type token for {@code DoFn<InputT, OutputT>.Context} given {@code InputT} and
- * {@code OutputT}.
+ * Generates a {@link TypeDescriptor} for {@code DoFn<InputT, OutputT>.Context} given {@code
+ * InputT} and {@code OutputT}.
*/
- private static <InputT, OutputT> TypeToken<DoFn<InputT, OutputT>.Context> doFnContextTypeOf(
- TypeToken<InputT> inputT, TypeToken<OutputT> outputT) {
- return new TypeToken<DoFn<InputT, OutputT>.Context>() {}.where(
+ private static <InputT, OutputT> TypeDescriptor<DoFn<InputT, OutputT>.Context> doFnContextTypeOf(
+ TypeDescriptor<InputT> inputT, TypeDescriptor<OutputT> outputT) {
+ return new TypeDescriptor<DoFn<InputT, OutputT>.Context>() {}.where(
new TypeParameter<InputT>() {}, inputT)
.where(new TypeParameter<OutputT>() {}, outputT);
}
- /** Generates a type token for {@code DoFn.InputProvider<InputT>} given {@code InputT}. */
- private static <InputT> TypeToken<DoFn.InputProvider<InputT>> inputProviderTypeOf(
- TypeToken<InputT> inputT) {
- return new TypeToken<DoFn.InputProvider<InputT>>() {}.where(
+ /**
+ * Generates a {@link TypeDescriptor} for {@code DoFn.InputProvider<InputT>} given {@code InputT}.
+ */
+ private static <InputT> TypeDescriptor<DoFn.InputProvider<InputT>> inputProviderTypeOf(
+ TypeDescriptor<InputT> inputT) {
+ return new TypeDescriptor<DoFn.InputProvider<InputT>>() {}.where(
new TypeParameter<InputT>() {}, inputT);
}
- /** Generates a type token for {@code DoFn.OutputReceiver<OutputT>} given {@code OutputT}. */
- private static <OutputT> TypeToken<DoFn.OutputReceiver<OutputT>> outputReceiverTypeOf(
- TypeToken<OutputT> inputT) {
- return new TypeToken<DoFn.OutputReceiver<OutputT>>() {}.where(
+ /**
+ * Generates a {@link TypeDescriptor} for {@code DoFn.OutputReceiver<OutputT>} given {@code
+ * OutputT}.
+ */
+ private static <OutputT> TypeDescriptor<DoFn.OutputReceiver<OutputT>> outputReceiverTypeOf(
+ TypeDescriptor<OutputT> inputT) {
+ return new TypeDescriptor<DoFn.OutputReceiver<OutputT>>() {}.where(
new TypeParameter<OutputT>() {}, inputT);
}
@VisibleForTesting
static DoFnSignature.ProcessElementMethod analyzeProcessElementMethod(
ErrorReporter errors,
- TypeToken<? extends DoFn<?, ?>> fnClass,
+ TypeDescriptor<? extends DoFn<?, ?>> fnClass,
Method m,
- TypeToken<?> inputT,
- TypeToken<?> outputT,
+ TypeDescriptor<?> inputT,
+ TypeDescriptor<?> outputT,
Map<String, StateDeclaration> stateDeclarations,
Map<String, TimerDeclaration> timerDeclarations) {
errors.checkArgument(
@@ -462,27 +466,27 @@ public class DoFnSignatures {
"Must return void or %s",
DoFn.ProcessContinuation.class.getSimpleName());
- TypeToken<?> processContextToken = doFnProcessContextTypeOf(inputT, outputT);
+ TypeDescriptor<?> processContextT = doFnProcessContextTypeOf(inputT, outputT);
Type[] params = m.getGenericParameterTypes();
- TypeToken<?> contextToken = null;
+ TypeDescriptor<?> contextT = null;
if (params.length > 0) {
- contextToken = fnClass.resolveType(params[0]);
+ contextT = fnClass.resolveType(params[0]);
}
errors.checkArgument(
- contextToken != null && contextToken.equals(processContextToken),
+ contextT != null && contextT.equals(processContextT),
"Must take %s as the first argument",
- formatType(processContextToken));
+ formatType(processContextT));
List<DoFnSignature.Parameter> extraParameters = new ArrayList<>();
Map<String, DoFnSignature.Parameter> stateParameters = new HashMap<>();
Map<String, DoFnSignature.Parameter> timerParameters = new HashMap<>();
- TypeToken<?> trackerT = null;
+ TypeDescriptor<?> trackerT = null;
- TypeToken<?> expectedInputProviderT = inputProviderTypeOf(inputT);
- TypeToken<?> expectedOutputReceiverT = outputReceiverTypeOf(outputT);
+ TypeDescriptor<?> expectedInputProviderT = inputProviderTypeOf(inputT);
+ TypeDescriptor<?> expectedOutputReceiverT = outputReceiverTypeOf(outputT);
for (int i = 1; i < params.length; ++i) {
- TypeToken<?> paramT = fnClass.resolveType(params[i]);
+ TypeDescriptor<?> paramT = fnClass.resolveType(params[i]);
Class<?> rawType = paramT.getRawType();
if (rawType.equals(BoundedWindow.class)) {
errors.checkArgument(
@@ -641,8 +645,8 @@ public class DoFnSignatures {
} else {
List<String> allowedParamTypes =
Arrays.asList(
- formatType(new TypeToken<BoundedWindow>() {}),
- formatType(new TypeToken<RestrictionTracker<?>>() {}));
+ formatType(new TypeDescriptor<BoundedWindow>() {}),
+ formatType(new TypeDescriptor<RestrictionTracker<?>>() {}));
errors.throwIllegalArgument(
"%s is not a valid context parameter. Should be one of %s",
formatType(paramT), allowedParamTypes);
@@ -665,17 +669,17 @@ public class DoFnSignatures {
@VisibleForTesting
static DoFnSignature.BundleMethod analyzeBundleMethod(
ErrorReporter errors,
- TypeToken<? extends DoFn<?, ?>> fnToken,
+ TypeDescriptor<? extends DoFn<?, ?>> fnT,
Method m,
- TypeToken<?> inputT,
- TypeToken<?> outputT) {
+ TypeDescriptor<?> inputT,
+ TypeDescriptor<?> outputT) {
errors.checkArgument(void.class.equals(m.getReturnType()), "Must return void");
- TypeToken<?> expectedContextToken = doFnContextTypeOf(inputT, outputT);
+ TypeDescriptor<?> expectedContextT = doFnContextTypeOf(inputT, outputT);
Type[] params = m.getGenericParameterTypes();
errors.checkArgument(
- params.length == 1 && fnToken.resolveType(params[0]).equals(expectedContextToken),
+ params.length == 1 && fnT.resolveType(params[0]).equals(expectedContextT),
"Must take a single argument of type %s",
- formatType(expectedContextToken));
+ formatType(expectedContextT));
return DoFnSignature.BundleMethod.create(m);
}
@@ -688,27 +692,33 @@ public class DoFnSignatures {
@VisibleForTesting
static DoFnSignature.GetInitialRestrictionMethod analyzeGetInitialRestrictionMethod(
- ErrorReporter errors, TypeToken<? extends DoFn> fnToken, Method m, TypeToken<?> inputT) {
+ ErrorReporter errors,
+ TypeDescriptor<? extends DoFn> fnT,
+ Method m,
+ TypeDescriptor<?> inputT) {
// Method is of the form:
// @GetInitialRestriction
// RestrictionT getInitialRestriction(InputT element);
Type[] params = m.getGenericParameterTypes();
errors.checkArgument(
- params.length == 1 && fnToken.resolveType(params[0]).equals(inputT),
+ params.length == 1 && fnT.resolveType(params[0]).equals(inputT),
"Must take a single argument of type %s",
formatType(inputT));
return DoFnSignature.GetInitialRestrictionMethod.create(
- m, fnToken.resolveType(m.getGenericReturnType()));
+ m, fnT.resolveType(m.getGenericReturnType()));
}
- /** Generates a type token for {@code List<T>} given {@code T}. */
- private static <T> TypeToken<List<T>> listTypeOf(TypeToken<T> elementT) {
- return new TypeToken<List<T>>() {}.where(new TypeParameter<T>() {}, elementT);
+ /** Generates a {@link TypeDescriptor} for {@code List<T>} given {@code T}. */
+ private static <T> TypeDescriptor<List<T>> listTypeOf(TypeDescriptor<T> elementT) {
+ return new TypeDescriptor<List<T>>() {}.where(new TypeParameter<T>() {}, elementT);
}
@VisibleForTesting
static DoFnSignature.SplitRestrictionMethod analyzeSplitRestrictionMethod(
- ErrorReporter errors, TypeToken<? extends DoFn> fnToken, Method m, TypeToken<?> inputT) {
+ ErrorReporter errors,
+ TypeDescriptor<? extends DoFn> fnT,
+ Method m,
+ TypeDescriptor<?> inputT) {
// Method is of the form:
// @SplitRestriction
// void splitRestriction(InputT element, RestrictionT restriction);
@@ -717,13 +727,13 @@ public class DoFnSignatures {
Type[] params = m.getGenericParameterTypes();
errors.checkArgument(params.length == 3, "Must have exactly 3 arguments");
errors.checkArgument(
- fnToken.resolveType(params[0]).equals(inputT),
+ fnT.resolveType(params[0]).equals(inputT),
"First argument must be the element type %s",
formatType(inputT));
- TypeToken<?> restrictionT = fnToken.resolveType(params[1]);
- TypeToken<?> receiverT = fnToken.resolveType(params[2]);
- TypeToken<?> expectedReceiverT = outputReceiverTypeOf(restrictionT);
+ TypeDescriptor<?> restrictionT = fnT.resolveType(params[1]);
+ TypeDescriptor<?> receiverT = fnT.resolveType(params[2]);
+ TypeDescriptor<?> expectedReceiverT = outputReceiverTypeOf(restrictionT);
errors.checkArgument(
receiverT.equals(expectedReceiverT),
"Third argument must be %s, but is %s",
@@ -777,45 +787,46 @@ public class DoFnSignatures {
}
}
- /** Generates a type token for {@code Coder<T>} given {@code T}. */
- private static <T> TypeToken<Coder<T>> coderTypeOf(TypeToken<T> elementT) {
- return new TypeToken<Coder<T>>() {}.where(new TypeParameter<T>() {}, elementT);
+ /** Generates a {@link TypeDescriptor} for {@code Coder<T>} given {@code T}. */
+ private static <T> TypeDescriptor<Coder<T>> coderTypeOf(TypeDescriptor<T> elementT) {
+ return new TypeDescriptor<Coder<T>>() {}.where(new TypeParameter<T>() {}, elementT);
}
@VisibleForTesting
static DoFnSignature.GetRestrictionCoderMethod analyzeGetRestrictionCoderMethod(
- ErrorReporter errors, TypeToken<? extends DoFn> fnToken, Method m) {
+ ErrorReporter errors, TypeDescriptor<? extends DoFn> fnT, Method m) {
errors.checkArgument(m.getParameterTypes().length == 0, "Must have zero arguments");
- TypeToken<?> resT = fnToken.resolveType(m.getGenericReturnType());
+ TypeDescriptor<?> resT = fnT.resolveType(m.getGenericReturnType());
errors.checkArgument(
- resT.isSubtypeOf(TypeToken.of(Coder.class)),
+ resT.isSubtypeOf(TypeDescriptor.of(Coder.class)),
"Must return a Coder, but returns %s",
formatType(resT));
return DoFnSignature.GetRestrictionCoderMethod.create(m, resT);
}
/**
- * Generates a type token for {@code RestrictionTracker<RestrictionT>} given {@code RestrictionT}.
+ * Generates a {@link TypeDescriptor} for {@code RestrictionTracker<RestrictionT>} given {@code
+ * RestrictionT}.
*/
private static <RestrictionT>
- TypeToken<RestrictionTracker<RestrictionT>> restrictionTrackerTypeOf(
- TypeToken<RestrictionT> restrictionT) {
- return new TypeToken<RestrictionTracker<RestrictionT>>() {}.where(
+ TypeDescriptor<RestrictionTracker<RestrictionT>> restrictionTrackerTypeOf(
+ TypeDescriptor<RestrictionT> restrictionT) {
+ return new TypeDescriptor<RestrictionTracker<RestrictionT>>() {}.where(
new TypeParameter<RestrictionT>() {}, restrictionT);
}
@VisibleForTesting
static DoFnSignature.NewTrackerMethod analyzeNewTrackerMethod(
- ErrorReporter errors, TypeToken<? extends DoFn> fnToken, Method m) {
+ ErrorReporter errors, TypeDescriptor<? extends DoFn> fnT, Method m) {
// Method is of the form:
// @NewTracker
// TrackerT newTracker(RestrictionT restriction);
Type[] params = m.getGenericParameterTypes();
errors.checkArgument(params.length == 1, "Must have a single argument");
- TypeToken<?> restrictionT = fnToken.resolveType(params[0]);
- TypeToken<?> trackerT = fnToken.resolveType(m.getGenericReturnType());
- TypeToken<?> expectedTrackerT = restrictionTrackerTypeOf(restrictionT);
+ TypeDescriptor<?> restrictionT = fnT.resolveType(params[0]);
+ TypeDescriptor<?> trackerT = fnT.resolveType(m.getGenericReturnType());
+ TypeDescriptor<?> expectedTrackerT = restrictionTrackerTypeOf(restrictionT);
errors.checkArgument(
trackerT.isSubtypeOf(expectedTrackerT),
"Returns %s, but must return a subtype of %s",
@@ -985,7 +996,7 @@ public class DoFnSignatures {
return ReflectHelpers.METHOD_FORMATTER.apply(method);
}
- private static String formatType(TypeToken<?> t) {
+ private static String formatType(TypeDescriptor<?> t) {
return ReflectHelpers.TYPE_SIMPLE_DESCRIPTION.apply(t.getType());
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8336b24c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
index 68278c5..573701b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
@@ -22,7 +22,6 @@ import static org.apache.beam.sdk.transforms.reflect.DoFnSignaturesTestUtils.err
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
-import com.google.common.reflect.TypeToken;
import java.util.List;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
@@ -34,6 +33,7 @@ import org.apache.beam.sdk.transforms.reflect.DoFnSignaturesTestUtils.FakeDoFn;
import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -403,12 +403,12 @@ public class DoFnSignaturesSplittableDoFnTest {
"Third argument must be OutputReceiver<SomeRestriction>, but is OutputReceiver<String>");
DoFnSignatures.analyzeSplitRestrictionMethod(
errors(),
- TypeToken.of(FakeDoFn.class),
+ TypeDescriptor.of(FakeDoFn.class),
new AnonymousMethod() {
void method(
Integer element, SomeRestriction restriction, DoFn.OutputReceiver<String> receiver) {}
}.getMethod(),
- TypeToken.of(Integer.class));
+ TypeDescriptor.of(Integer.class));
}
@Test
@@ -422,14 +422,14 @@ public class DoFnSignaturesSplittableDoFnTest {
thrown.expectMessage("First argument must be the element type Integer");
DoFnSignatures.analyzeSplitRestrictionMethod(
errors(),
- TypeToken.of(FakeDoFn.class),
+ TypeDescriptor.of(FakeDoFn.class),
new AnonymousMethod() {
void method(
String element,
SomeRestriction restriction,
DoFn.OutputReceiver<SomeRestriction> receiver) {}
}.getMethod(),
- TypeToken.of(Integer.class));
+ TypeDescriptor.of(Integer.class));
}
@Test
@@ -437,7 +437,7 @@ public class DoFnSignaturesSplittableDoFnTest {
thrown.expectMessage("Must have exactly 3 arguments");
DoFnSignatures.analyzeSplitRestrictionMethod(
errors(),
- TypeToken.of(FakeDoFn.class),
+ TypeDescriptor.of(FakeDoFn.class),
new AnonymousMethod() {
private void method(
Integer element,
@@ -445,7 +445,7 @@ public class DoFnSignaturesSplittableDoFnTest {
DoFn.OutputReceiver<SomeRestriction> receiver,
Object extra) {}
}.getMethod(),
- TypeToken.of(Integer.class));
+ TypeDescriptor.of(Integer.class));
}
@Test
@@ -519,7 +519,7 @@ public class DoFnSignaturesSplittableDoFnTest {
thrown.expectMessage("Must have a single argument");
DoFnSignatures.analyzeNewTrackerMethod(
errors(),
- TypeToken.of(FakeDoFn.class),
+ TypeDescriptor.of(FakeDoFn.class),
new AnonymousMethod() {
private SomeRestrictionTracker method(SomeRestriction restriction, Object extra) {
return null;
@@ -533,7 +533,7 @@ public class DoFnSignaturesSplittableDoFnTest {
"Returns SomeRestrictionTracker, but must return a subtype of RestrictionTracker<String>");
DoFnSignatures.analyzeNewTrackerMethod(
errors(),
- TypeToken.of(FakeDoFn.class),
+ TypeDescriptor.of(FakeDoFn.class),
new AnonymousMethod() {
private SomeRestrictionTracker method(String restriction) {
return null;
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8336b24c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
index fe88c3b..52ecb2a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
@@ -25,7 +25,6 @@ import static org.hamcrest.Matchers.not;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
-import com.google.common.reflect.TypeToken;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
@@ -66,12 +65,12 @@ public class DoFnSignaturesTest {
DoFnSignatures.analyzeBundleMethod(
errors(),
- TypeToken.of(FakeDoFn.class),
+ TypeDescriptor.of(FakeDoFn.class),
new DoFnSignaturesTestUtils.AnonymousMethod() {
void method(DoFn<Integer, String>.Context c, int n) {}
}.getMethod(),
- TypeToken.of(Integer.class),
- TypeToken.of(String.class));
+ TypeDescriptor.of(Integer.class),
+ TypeDescriptor.of(String.class));
}
@Test
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8336b24c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java
index ce00f2d..49e2ba7 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java
@@ -17,11 +17,11 @@
*/
package org.apache.beam.sdk.transforms.reflect;
-import com.google.common.reflect.TypeToken;
import java.lang.reflect.Method;
import java.util.Collections;
import java.util.NoSuchElementException;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.values.TypeDescriptor;
/** Utilities for use in {@link DoFnSignatures} tests. */
class DoFnSignaturesTestUtils {
@@ -57,10 +57,10 @@ class DoFnSignaturesTestUtils {
throws Exception {
return DoFnSignatures.analyzeProcessElementMethod(
errors(),
- TypeToken.of(FakeDoFn.class),
+ TypeDescriptor.of(FakeDoFn.class),
method.getMethod(),
- TypeToken.of(Integer.class),
- TypeToken.of(String.class),
+ TypeDescriptor.of(Integer.class),
+ TypeDescriptor.of(String.class),
Collections.EMPTY_MAP,
Collections.EMPTY_MAP);
}
[10/50] incubator-beam git commit: [BEAM-790] Validate
PipelineOptions Default annotation
Posted by th...@apache.org.
[BEAM-790] Validate PipelineOptions Default annotation
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f605b02a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f605b02a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f605b02a
Branch: refs/heads/apex-runner
Commit: f605b02a72258aee20d607e54a443113bfb762ad
Parents: 6a05cf4
Author: Pei He <pe...@google.com>
Authored: Thu Oct 20 18:07:35 2016 -0700
Committer: Luke Cwik <lc...@google.com>
Committed: Thu Nov 3 11:38:42 2016 -0700
----------------------------------------------------------------------
.../sdk/options/PipelineOptionsFactory.java | 248 ++++++++++++++-----
.../sdk/options/ProxyInvocationHandler.java | 5 +-
.../beam/sdk/util/common/ReflectHelpers.java | 17 ++
.../sdk/options/PipelineOptionsFactoryTest.java | 239 ++++++++++++++++++
.../sdk/options/ProxyInvocationHandlerTest.java | 29 ++-
.../sdk/util/common/ReflectHelpersTest.java | 25 ++
6 files changed, 491 insertions(+), 72 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f605b02a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index eefe8c7..304e166 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -1057,56 +1057,130 @@ public class PipelineOptionsFactory {
methodNameToAllMethodMap.put(method, method);
}
- List<InconsistentlyIgnoredGetters> incompletelyIgnoredGetters = new ArrayList<>();
- List<IgnoredSetter> ignoredSetters = new ArrayList<>();
+ // Verify that there is no getter with a mixed @JsonIgnore annotation.
+ validateGettersHaveConsistentAnnotation(
+ methodNameToAllMethodMap, descriptors, AnnotationPredicates.JSON_IGNORE);
- for (PropertyDescriptor descriptor : descriptors) {
+ // Verify that there is no getter with a mixed @Default annotation.
+ validateGettersHaveConsistentAnnotation(
+ methodNameToAllMethodMap, descriptors, AnnotationPredicates.DEFAULT_VALUE);
+
+ // Verify that no setter has @JsonIgnore.
+ validateSettersDoNotHaveAnnotation(
+ methodNameToAllMethodMap, descriptors, AnnotationPredicates.JSON_IGNORE);
+
+ // Verify that no setter has @Default.
+ validateSettersDoNotHaveAnnotation(
+ methodNameToAllMethodMap, descriptors, AnnotationPredicates.DEFAULT_VALUE);
+ }
+
+ /**
+ * Validates that getters don't have mixed annotation.
+ */
+ private static void validateGettersHaveConsistentAnnotation(
+ SortedSetMultimap<Method, Method> methodNameToAllMethodMap,
+ List<PropertyDescriptor> descriptors,
+ final AnnotationPredicates annotationPredicates) {
+ List<InconsistentlyAnnotatedGetters> inconsistentlyAnnotatedGetters = new ArrayList<>();
+ for (final PropertyDescriptor descriptor : descriptors) {
if (descriptor.getReadMethod() == null
- || descriptor.getWriteMethod() == null
- || IGNORED_METHODS.contains(descriptor.getReadMethod())
- || IGNORED_METHODS.contains(descriptor.getWriteMethod())) {
+ || IGNORED_METHODS.contains(descriptor.getReadMethod())) {
continue;
}
- // Verify that there is no getter with a mixed @JsonIgnore annotation and verify
- // that no setter has @JsonIgnore.
+
SortedSet<Method> getters = methodNameToAllMethodMap.get(descriptor.getReadMethod());
- SortedSet<Method> gettersWithJsonIgnore = Sets.filter(getters, JsonIgnorePredicate.INSTANCE);
+ SortedSet<Method> gettersWithTheAnnotation =
+ Sets.filter(getters, annotationPredicates.forMethod);
+ Set<Annotation> distinctAnnotations = Sets.newLinkedHashSet(FluentIterable
+ .from(gettersWithTheAnnotation)
+ .transformAndConcat(new Function<Method, Iterable<? extends Annotation>>() {
+ @Nonnull
+ @Override
+ public Iterable<? extends Annotation> apply(@Nonnull Method method) {
+ return FluentIterable.of(method.getAnnotations());
+ }
+ })
+ .filter(annotationPredicates.forAnnotation));
+
+
+ if (distinctAnnotations.size() > 1) {
+ throw new IllegalArgumentException(String.format(
+ "Property [%s] is marked with contradictory annotations. Found [%s].",
+ descriptor.getName(),
+ FluentIterable.from(gettersWithTheAnnotation)
+ .transformAndConcat(new Function<Method, Iterable<String>>() {
+ @Nonnull
+ @Override
+ public Iterable<String> apply(final @Nonnull Method method) {
+ return FluentIterable.of(method.getAnnotations())
+ .filter(annotationPredicates.forAnnotation)
+ .transform(new Function<Annotation, String>() {
+ @Nonnull
+ @Override
+ public String apply(@Nonnull Annotation annotation) {
+ return String.format(
+ "[%s on %s]",
+ ReflectHelpers.ANNOTATION_FORMATTER.apply(annotation),
+ ReflectHelpers.CLASS_AND_METHOD_FORMATTER.apply(method));
+ }
+ });
+
+ }
+ })
+ .join(Joiner.on(", "))));
+ }
Iterable<String> getterClassNames = FluentIterable.from(getters)
.transform(MethodToDeclaringClassFunction.INSTANCE)
.transform(ReflectHelpers.CLASS_NAME);
- Iterable<String> gettersWithJsonIgnoreClassNames = FluentIterable.from(gettersWithJsonIgnore)
+ Iterable<String> gettersWithTheAnnotationClassNames =
+ FluentIterable.from(gettersWithTheAnnotation)
.transform(MethodToDeclaringClassFunction.INSTANCE)
.transform(ReflectHelpers.CLASS_NAME);
- if (!(gettersWithJsonIgnore.isEmpty() || getters.size() == gettersWithJsonIgnore.size())) {
- InconsistentlyIgnoredGetters err = new InconsistentlyIgnoredGetters();
+ if (!(gettersWithTheAnnotation.isEmpty()
+ || getters.size() == gettersWithTheAnnotation.size())) {
+ InconsistentlyAnnotatedGetters err = new InconsistentlyAnnotatedGetters();
err.descriptor = descriptor;
err.getterClassNames = getterClassNames;
- err.gettersWithJsonIgnoreClassNames = gettersWithJsonIgnoreClassNames;
- incompletelyIgnoredGetters.add(err);
+ err.gettersWithTheAnnotationClassNames = gettersWithTheAnnotationClassNames;
+ inconsistentlyAnnotatedGetters.add(err);
}
- if (!incompletelyIgnoredGetters.isEmpty()) {
+ }
+ throwForGettersWithInconsistentAnnotation(
+ inconsistentlyAnnotatedGetters, annotationPredicates.annotationClass);
+ }
+
+ /**
+ * Validates that setters don't have the given annotation.
+ */
+ private static void validateSettersDoNotHaveAnnotation(
+ SortedSetMultimap<Method, Method> methodNameToAllMethodMap,
+ List<PropertyDescriptor> descriptors,
+ AnnotationPredicates annotationPredicates) {
+ List<AnnotatedSetter> annotatedSetters = new ArrayList<>();
+ for (PropertyDescriptor descriptor : descriptors) {
+ if (descriptor.getWriteMethod() == null
+ || IGNORED_METHODS.contains(descriptor.getWriteMethod())) {
continue;
}
+ SortedSet<Method> settersWithTheAnnotation = Sets.filter(
+ methodNameToAllMethodMap.get(descriptor.getWriteMethod()),
+ annotationPredicates.forMethod);
- SortedSet<Method> settersWithJsonIgnore =
- Sets.filter(methodNameToAllMethodMap.get(descriptor.getWriteMethod()),
- JsonIgnorePredicate.INSTANCE);
-
- Iterable<String> settersWithJsonIgnoreClassNames = FluentIterable.from(settersWithJsonIgnore)
+ Iterable<String> settersWithTheAnnotationClassNames =
+ FluentIterable.from(settersWithTheAnnotation)
.transform(MethodToDeclaringClassFunction.INSTANCE)
.transform(ReflectHelpers.CLASS_NAME);
- if (!settersWithJsonIgnore.isEmpty()) {
- IgnoredSetter ignored = new IgnoredSetter();
- ignored.descriptor = descriptor;
- ignored.settersWithJsonIgnoreClassNames = settersWithJsonIgnoreClassNames;
- ignoredSetters.add(ignored);
+ if (!settersWithTheAnnotation.isEmpty()) {
+ AnnotatedSetter annotated = new AnnotatedSetter();
+ annotated.descriptor = descriptor;
+ annotated.settersWithTheAnnotationClassNames = settersWithTheAnnotationClassNames;
+ annotatedSetters.add(annotated);
}
}
- throwForGettersWithInconsistentJsonIgnore(incompletelyIgnoredGetters);
- throwForSettersWithJsonIgnore(ignoredSetters);
+ throwForSettersWithTheAnnotation(annotatedSetters, annotationPredicates.annotationClass);
}
/**
@@ -1221,53 +1295,62 @@ public class PipelineOptionsFactory {
}
}
- private static class InconsistentlyIgnoredGetters {
+ private static class InconsistentlyAnnotatedGetters {
PropertyDescriptor descriptor;
Iterable<String> getterClassNames;
- Iterable<String> gettersWithJsonIgnoreClassNames;
+ Iterable<String> gettersWithTheAnnotationClassNames;
}
- private static void throwForGettersWithInconsistentJsonIgnore(
- List<InconsistentlyIgnoredGetters> getters) {
+ private static void throwForGettersWithInconsistentAnnotation(
+ List<InconsistentlyAnnotatedGetters> getters,
+ Class<? extends Annotation> annotationClass) {
if (getters.size() == 1) {
- InconsistentlyIgnoredGetters getter = getters.get(0);
+ InconsistentlyAnnotatedGetters getter = getters.get(0);
throw new IllegalArgumentException(String.format(
- "Expected getter for property [%s] to be marked with @JsonIgnore on all %s, "
+ "Expected getter for property [%s] to be marked with @%s on all %s, "
+ "found only on %s",
- getter.descriptor.getName(), getter.getterClassNames,
- getter.gettersWithJsonIgnoreClassNames));
+ getter.descriptor.getName(),
+ annotationClass.getSimpleName(),
+ getter.getterClassNames,
+ getter.gettersWithTheAnnotationClassNames));
} else if (getters.size() > 1) {
- StringBuilder errorBuilder =
- new StringBuilder("Property getters are inconsistently marked with @JsonIgnore:");
- for (InconsistentlyIgnoredGetters getter : getters) {
+ StringBuilder errorBuilder = new StringBuilder(String.format(
+ "Property getters are inconsistently marked with @%s:", annotationClass.getSimpleName()));
+ for (InconsistentlyAnnotatedGetters getter : getters) {
errorBuilder.append(
String.format("%n - Expected for property [%s] to be marked on all %s, "
+ "found only on %s",
getter.descriptor.getName(), getter.getterClassNames,
- getter.gettersWithJsonIgnoreClassNames));
+ getter.gettersWithTheAnnotationClassNames));
}
throw new IllegalArgumentException(errorBuilder.toString());
}
}
- private static class IgnoredSetter {
+ private static class AnnotatedSetter {
PropertyDescriptor descriptor;
- Iterable<String> settersWithJsonIgnoreClassNames;
+ Iterable<String> settersWithTheAnnotationClassNames;
}
- private static void throwForSettersWithJsonIgnore(List<IgnoredSetter> setters) {
+ private static void throwForSettersWithTheAnnotation(
+ List<AnnotatedSetter> setters,
+ Class<? extends Annotation> annotationClass) {
if (setters.size() == 1) {
- IgnoredSetter setter = setters.get(0);
- throw new IllegalArgumentException(
- String.format("Expected setter for property [%s] to not be marked with @JsonIgnore on %s",
- setter.descriptor.getName(), setter.settersWithJsonIgnoreClassNames));
+ AnnotatedSetter setter = setters.get(0);
+ throw new IllegalArgumentException(String.format(
+ "Expected setter for property [%s] to not be marked with @%s on %s",
+ setter.descriptor.getName(),
+ annotationClass.getSimpleName(),
+ setter.settersWithTheAnnotationClassNames));
} else if (setters.size() > 1) {
- StringBuilder builder = new StringBuilder("Found setters marked with @JsonIgnore:");
- for (IgnoredSetter setter : setters) {
- builder.append(
- String.format("%n - Setter for property [%s] should not be marked with @JsonIgnore "
- + "on %s",
- setter.descriptor.getName(), setter.settersWithJsonIgnoreClassNames));
+ StringBuilder builder = new StringBuilder(
+ String.format("Found setters marked with @%s:", annotationClass.getSimpleName()));
+ for (AnnotatedSetter setter : setters) {
+ builder.append(String.format(
+ "%n - Setter for property [%s] should not be marked with @%s on %s",
+ setter.descriptor.getName(),
+ annotationClass.getSimpleName(),
+ setter.settersWithTheAnnotationClassNames));
}
throw new IllegalArgumentException(builder.toString());
}
@@ -1353,14 +1436,61 @@ public class PipelineOptionsFactory {
}
/**
- * A {@link Predicate} that returns true if the method is annotated with
- * {@link JsonIgnore @JsonIgnore}.
+ * A {@link Predicate} that returns true if the method is annotated with {@code annotationClass}.
*/
- static class JsonIgnorePredicate implements Predicate<Method> {
- static final JsonIgnorePredicate INSTANCE = new JsonIgnorePredicate();
- @Override
- public boolean apply(Method input) {
- return input.isAnnotationPresent(JsonIgnore.class);
+ static class AnnotationPredicates {
+ static final AnnotationPredicates JSON_IGNORE = new AnnotationPredicates(
+ JsonIgnore.class,
+ new Predicate<Annotation>() {
+ @Override
+ public boolean apply(@Nonnull Annotation input) {
+ return JsonIgnore.class.equals(input.annotationType());
+ }
+ },
+ new Predicate<Method>() {
+ @Override
+ public boolean apply(@Nonnull Method input) {
+ return input.isAnnotationPresent(JsonIgnore.class);
+ }});
+
+ private static final Set<Class<?>> DEFAULT_ANNOTATION_CLASSES = Sets.newHashSet(
+ FluentIterable.of(Default.class.getDeclaredClasses())
+ .filter(new Predicate<Class<?>>() {
+ @Override
+ public boolean apply(@Nonnull Class<?> klass) {
+ return klass.isAnnotation();
+ }}));
+
+ static final AnnotationPredicates DEFAULT_VALUE = new AnnotationPredicates(
+ Default.class,
+ new Predicate<Annotation>() {
+ @Override
+ public boolean apply(@Nonnull Annotation input) {
+ return DEFAULT_ANNOTATION_CLASSES.contains(input.annotationType());
+ }
+ },
+ new Predicate<Method> () {
+ @Override
+ public boolean apply(@Nonnull Method input) {
+ for (Annotation annotation : input.getAnnotations()) {
+ if (DEFAULT_ANNOTATION_CLASSES.contains(annotation.annotationType())) {
+ return true;
+ }
+ }
+ return false;
+ }});
+
+ final Class<? extends Annotation> annotationClass;
+ final Predicate<Annotation> forAnnotation;
+ final Predicate<Method> forMethod;
+
+ AnnotationPredicates(
+ Class<? extends Annotation> annotationClass,
+ Predicate<Annotation> forAnnotation,
+ Predicate<Method> forMethod) {
+ this.annotationClass = annotationClass;
+ this.forAnnotation = forAnnotation;
+ this.forMethod = forMethod;
}
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f605b02a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
index f7c6da2..86f9918 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
@@ -63,7 +63,7 @@ import java.util.TreeMap;
import java.util.concurrent.ThreadLocalRandom;
import javax.annotation.Nullable;
import javax.annotation.concurrent.ThreadSafe;
-import org.apache.beam.sdk.options.PipelineOptionsFactory.JsonIgnorePredicate;
+import org.apache.beam.sdk.options.PipelineOptionsFactory.AnnotationPredicates;
import org.apache.beam.sdk.options.PipelineOptionsFactory.Registration;
import org.apache.beam.sdk.options.ValueProvider.RuntimeValueProvider;
import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
@@ -638,7 +638,8 @@ class ProxyInvocationHandler implements InvocationHandler {
// Find all the method names that are annotated with JSON ignore.
Set<String> jsonIgnoreMethodNames = FluentIterable.from(
ReflectHelpers.getClosureOfMethodsOnInterfaces(interfaces))
- .filter(JsonIgnorePredicate.INSTANCE).transform(new Function<Method, String>() {
+ .filter(AnnotationPredicates.JSON_IGNORE.forMethod)
+ .transform(new Function<Method, String>() {
@Override
public String apply(Method input) {
return input.getName();
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f605b02a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java
index 2d92162..2b08fee 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java
@@ -26,6 +26,7 @@ import com.google.common.base.Joiner;
import com.google.common.collect.FluentIterable;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Queues;
+import java.lang.annotation.Annotation;
import java.lang.reflect.GenericArrayType;
import java.lang.reflect.Method;
import java.lang.reflect.ParameterizedType;
@@ -86,6 +87,22 @@ public class ReflectHelpers {
}
};
+ /**
+ * A {@link Function} that returns a concise string for a {@link Annotation}.
+ */
+ public static final Function<Annotation, String> ANNOTATION_FORMATTER =
+ new Function<Annotation, String>() {
+ @Override
+ public String apply(@Nonnull Annotation annotation) {
+ String annotationName = annotation.annotationType().getName();
+ String annotationNameWithoutPackage =
+ annotationName.substring(annotationName.lastIndexOf('.') + 1).replace('$', '.');
+ String annotationToString = annotation.toString();
+ String values = annotationToString.substring(annotationToString.indexOf('('));
+ return String.format("%s%s", annotationNameWithoutPackage, values);
+ }
+ };
+
/** A {@link Function} that formats types. */
public static final Function<Type, String> TYPE_SIMPLE_DESCRIPTION =
new Function<Type, String>() {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f605b02a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java
index 64d8c51..0a2324f 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java
@@ -369,6 +369,16 @@ public class PipelineOptionsFactoryTest {
void setObject(Object value);
}
+ /**
+ * This class is has a conflicting {@link JsonIgnore @JsonIgnore} value with
+ * {@link GetterWithJsonIgnore}.
+ */
+ public interface GetterWithInconsistentJsonIgnoreValue extends PipelineOptions {
+ @JsonIgnore(value = false)
+ Object getObject();
+ void setObject(Object value);
+ }
+
@Test
public void testNotAllGettersAnnotatedWithJsonIgnore() throws Exception {
// Initial construction is valid.
@@ -441,6 +451,235 @@ public class PipelineOptionsFactoryTest {
options.as(MultipleGettersWithInconsistentJsonIgnore.class);
}
+ /** Test interface that has {@link Default @Default} on a setter for a property. */
+ public interface SetterWithDefault extends PipelineOptions {
+ String getValue();
+ @Default.String("abc")
+ void setValue(String value);
+ }
+
+ @Test
+ public void testSetterAnnotatedWithDefault() throws Exception {
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage(
+ "Expected setter for property [value] to not be marked with @Default on ["
+ + "org.apache.beam.sdk.options.PipelineOptionsFactoryTest$SetterWithDefault]");
+ PipelineOptionsFactory.as(SetterWithDefault.class);
+ }
+
+ /** Test interface that has {@link Default @Default} on multiple setters. */
+ public interface MultiSetterWithDefault extends SetterWithDefault {
+ Integer getOther();
+ @Default.String("abc")
+ void setOther(Integer other);
+ }
+
+ @Test
+ public void testMultipleSettersAnnotatedWithDefault() throws Exception {
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage("Found setters marked with @Default:");
+ expectedException.expectMessage(
+ "property [other] should not be marked with @Default on ["
+ + "org.apache.beam.sdk.options.PipelineOptionsFactoryTest$MultiSetterWithDefault]");
+ expectedException.expectMessage(
+ "property [value] should not be marked with @Default on ["
+ + "org.apache.beam.sdk.options.PipelineOptionsFactoryTest$SetterWithDefault]");
+ PipelineOptionsFactory.as(MultiSetterWithDefault.class);
+ }
+
+ /**
+ * This class is has a conflicting field with {@link CombinedObject} that doesn't have
+ * {@link Default @Default}.
+ */
+ private interface GetterWithDefault extends PipelineOptions {
+ @Default.Integer(1)
+ Object getObject();
+ void setObject(Object value);
+ }
+
+ /**
+ * This class is consistent with {@link GetterWithDefault} that has the same
+ * {@link Default @Default}.
+ */
+ private interface GetterWithConsistentDefault extends PipelineOptions {
+ @Default.Integer(1)
+ Object getObject();
+ void setObject(Object value);
+ }
+
+ /**
+ * This class is inconsistent with {@link GetterWithDefault} that has a different
+ * {@link Default @Default}.
+ */
+ private interface GetterWithInconsistentDefaultType extends PipelineOptions {
+ @Default.String("abc")
+ Object getObject();
+ void setObject(Object value);
+ }
+
+ /**
+ * This class is inconsistent with {@link GetterWithDefault} that has a different
+ * {@link Default @Default} value.
+ */
+ private interface GetterWithInconsistentDefaultValue extends PipelineOptions {
+ @Default.Integer(0)
+ Object getObject();
+ void setObject(Object value);
+ }
+
+ @Test
+ public void testNotAllGettersAnnotatedWithDefault() throws Exception {
+ // Initial construction is valid.
+ GetterWithDefault options = PipelineOptionsFactory.as(GetterWithDefault.class);
+
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage(
+ "Expected getter for property [object] to be marked with @Default on all ["
+ + "org.apache.beam.sdk.options.PipelineOptionsFactoryTest$GetterWithDefault, "
+ + "org.apache.beam.sdk.options.PipelineOptionsFactoryTest$MissingSetter], "
+ + "found only on [org.apache.beam.sdk.options."
+ + "PipelineOptionsFactoryTest$GetterWithDefault]");
+
+ // When we attempt to convert, we should error at this moment.
+ options.as(CombinedObject.class);
+ }
+
+ @Test
+ public void testGettersAnnotatedWithConsistentDefault() throws Exception {
+ GetterWithConsistentDefault options = PipelineOptionsFactory
+ .as(GetterWithDefault.class)
+ .as(GetterWithConsistentDefault.class);
+
+ assertEquals(1, options.getObject());
+ }
+
+ @Test
+ public void testGettersAnnotatedWithInconsistentDefault() throws Exception {
+ // Initial construction is valid.
+ GetterWithDefault options = PipelineOptionsFactory.as(GetterWithDefault.class);
+
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage(
+ "Property [object] is marked with contradictory annotations. Found ["
+ + "[Default.Integer(value=1) on org.apache.beam.sdk.options.PipelineOptionsFactoryTest"
+ + "$GetterWithDefault#getObject()], "
+ + "[Default.String(value=abc) on org.apache.beam.sdk.options.PipelineOptionsFactoryTest"
+ + "$GetterWithInconsistentDefaultType#getObject()]].");
+
+ // When we attempt to convert, we should error at this moment.
+ options.as(GetterWithInconsistentDefaultType.class);
+ }
+
+ @Test
+ public void testGettersAnnotatedWithInconsistentDefaultValue() throws Exception {
+ // Initial construction is valid.
+ GetterWithDefault options = PipelineOptionsFactory.as(GetterWithDefault.class);
+
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage(
+ "Property [object] is marked with contradictory annotations. Found ["
+ + "[Default.Integer(value=1) on org.apache.beam.sdk.options.PipelineOptionsFactoryTest"
+ + "$GetterWithDefault#getObject()], "
+ + "[Default.Integer(value=0) on org.apache.beam.sdk.options.PipelineOptionsFactoryTest"
+ + "$GetterWithInconsistentDefaultValue#getObject()]].");
+
+ // When we attempt to convert, we should error at this moment.
+ options.as(GetterWithInconsistentDefaultValue.class);
+ }
+
+ @Test
+ public void testGettersAnnotatedWithInconsistentJsonIgnoreValue() throws Exception {
+ // Initial construction is valid.
+ GetterWithJsonIgnore options = PipelineOptionsFactory.as(GetterWithJsonIgnore.class);
+
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage(
+ "Property [object] is marked with contradictory annotations. Found ["
+ + "[JsonIgnore(value=false) on org.apache.beam.sdk.options.PipelineOptionsFactoryTest"
+ + "$GetterWithInconsistentJsonIgnoreValue#getObject()], "
+ + "[JsonIgnore(value=true) on org.apache.beam.sdk.options.PipelineOptionsFactoryTest"
+ + "$GetterWithJsonIgnore#getObject()]].");
+
+ // When we attempt to convert, we should error at this moment.
+ options.as(GetterWithInconsistentJsonIgnoreValue.class);
+ }
+
+ private interface GettersWithMultipleDefault extends PipelineOptions {
+ @Default.String("abc")
+ @Default.Integer(0)
+ Object getObject();
+ void setObject(Object value);
+ }
+
+ @Test
+ public void testGettersWithMultipleDefaults() throws Exception {
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage(
+ "Property [object] is marked with contradictory annotations. Found ["
+ + "[Default.String(value=abc) on org.apache.beam.sdk.options.PipelineOptionsFactoryTest"
+ + "$GettersWithMultipleDefault#getObject()], "
+ + "[Default.Integer(value=0) on org.apache.beam.sdk.options.PipelineOptionsFactoryTest"
+ + "$GettersWithMultipleDefault#getObject()]].");
+
+ // When we attempt to create, we should error at this moment.
+ PipelineOptionsFactory.as(GettersWithMultipleDefault.class);
+ }
+
+ private interface MultiGettersWithDefault extends PipelineOptions {
+ Object getObject();
+ void setObject(Object value);
+
+ @Default.Integer(1)
+ Integer getOther();
+ void setOther(Integer value);
+
+ Void getConsistent();
+ void setConsistent(Void consistent);
+ }
+
+ private interface MultipleGettersWithInconsistentDefault extends PipelineOptions {
+ @Default.Boolean(true)
+ Object getObject();
+ void setObject(Object value);
+
+ Integer getOther();
+ void setOther(Integer value);
+
+ Void getConsistent();
+ void setConsistent(Void consistent);
+ }
+
+ @Test
+ public void testMultipleGettersWithInconsistentDefault() {
+ // Initial construction is valid.
+ MultiGettersWithDefault options = PipelineOptionsFactory.as(MultiGettersWithDefault.class);
+
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage("Property getters are inconsistently marked with @Default:");
+ expectedException.expectMessage(
+ "property [object] to be marked on all");
+ expectedException.expectMessage("found only on [org.apache.beam.sdk.options."
+ + "PipelineOptionsFactoryTest$MultiGettersWithDefault]");
+ expectedException.expectMessage(
+ "property [other] to be marked on all");
+ expectedException.expectMessage("found only on [org.apache.beam.sdk.options."
+ + "PipelineOptionsFactoryTest$MultipleGettersWithInconsistentDefault]");
+
+ expectedException.expectMessage(Matchers.anyOf(
+ containsString(java.util.Arrays.toString(new String[]
+ {"org.apache.beam.sdk.options."
+ + "PipelineOptionsFactoryTest$MultipleGettersWithInconsistentDefault",
+ "org.apache.beam.sdk.options.PipelineOptionsFactoryTest$MultiGettersWithDefault"})),
+ containsString(java.util.Arrays.toString(new String[]
+ {"org.apache.beam.sdk.options.PipelineOptionsFactoryTest$MultiGettersWithDefault",
+ "org.apache.beam.sdk.options."
+ + "PipelineOptionsFactoryTest$MultipleGettersWithInconsistentDefault"}))));
+ expectedException.expectMessage(not(containsString("property [consistent]")));
+
+ // When we attempt to convert, we should error immediately
+ options.as(MultipleGettersWithInconsistentDefault.class);
+ }
+
@Test
public void testAppNameIsNotOverriddenWhenPassedInViaCommandLine() {
ApplicationNameOptions options = PipelineOptionsFactory
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f605b02a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java
index 52b98ee..5e97eed 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java
@@ -235,38 +235,45 @@ public class ProxyInvocationHandlerTest {
.testEquals();
}
+ /** A test interface for string with default. */
+ public interface StringWithDefault extends PipelineOptions {
+ @Default.String("testString")
+ String getString();
+ void setString(String value);
+ }
+
@Test
public void testToString() throws Exception {
ProxyInvocationHandler handler = new ProxyInvocationHandler(Maps.<String, Object>newHashMap());
- Simple proxy = handler.as(Simple.class);
+ StringWithDefault proxy = handler.as(StringWithDefault.class);
proxy.setString("stringValue");
DefaultAnnotations proxy2 = proxy.as(DefaultAnnotations.class);
proxy2.setLong(57L);
assertEquals("Current Settings:\n"
- + " long: 57\n"
- + " string: stringValue\n",
+ + " long: 57\n"
+ + " string: stringValue\n",
proxy.toString());
}
@Test
public void testToStringAfterDeserializationContainsJsonEntries() throws Exception {
ProxyInvocationHandler handler = new ProxyInvocationHandler(Maps.<String, Object>newHashMap());
- Simple proxy = handler.as(Simple.class);
+ StringWithDefault proxy = handler.as(StringWithDefault.class);
Long optionsId = proxy.getOptionsId();
proxy.setString("stringValue");
DefaultAnnotations proxy2 = proxy.as(DefaultAnnotations.class);
proxy2.setLong(57L);
assertEquals(String.format("Current Settings:\n"
- + " long: 57\n"
- + " optionsId: %d\n"
- + " string: \"stringValue\"\n", optionsId),
+ + " long: 57\n"
+ + " optionsId: %d\n"
+ + " string: \"stringValue\"\n", optionsId),
serializeDeserialize(PipelineOptions.class, proxy2).toString());
}
@Test
public void testToStringAfterDeserializationContainsOverriddenEntries() throws Exception {
ProxyInvocationHandler handler = new ProxyInvocationHandler(Maps.<String, Object>newHashMap());
- Simple proxy = handler.as(Simple.class);
+ StringWithDefault proxy = handler.as(StringWithDefault.class);
Long optionsId = proxy.getOptionsId();
proxy.setString("stringValue");
DefaultAnnotations proxy2 = proxy.as(DefaultAnnotations.class);
@@ -274,9 +281,9 @@ public class ProxyInvocationHandlerTest {
Simple deserializedOptions = serializeDeserialize(Simple.class, proxy2);
deserializedOptions.setString("overriddenValue");
assertEquals(String.format("Current Settings:\n"
- + " long: 57\n"
- + " optionsId: %d\n"
- + " string: overriddenValue\n", optionsId),
+ + " long: 57\n"
+ + " optionsId: %d\n"
+ + " string: overriddenValue\n", optionsId),
deserializedOptions.toString());
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f605b02a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java
index e1073da..8a1708c 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java
@@ -19,8 +19,11 @@ package org.apache.beam.sdk.util.common;
import static org.junit.Assert.assertEquals;
+import com.fasterxml.jackson.annotation.JsonIgnore;
import java.util.List;
import java.util.Map;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -123,4 +126,26 @@ public class ReflectHelpersTest {
ReflectHelpers.TYPE_SIMPLE_DESCRIPTION.apply(
new TypeDescriptor<Map<? super InputT, ? extends OutputT>>() {}.getType()));
}
+
+ private interface Options extends PipelineOptions {
+ @Default.String("package.OuterClass$InnerClass#method()")
+ String getString();
+
+ @JsonIgnore
+ Object getObject();
+ }
+
+ @Test
+ public void testAnnotationFormatter() throws Exception {
+ assertEquals(
+ "Default.String(value=package.OuterClass$InnerClass#method())",
+ ReflectHelpers.ANNOTATION_FORMATTER.apply(
+ Options.class.getMethod("getString").getAnnotations()[0]));
+
+ assertEquals(
+ "JsonIgnore(value=true)",
+ ReflectHelpers.ANNOTATION_FORMATTER.apply(
+ Options.class.getMethod("getObject").getAnnotations()[0]));
+ }
+
}
[25/50] incubator-beam git commit: DoFnSignature: Make
TypeDescriptor-returning methods public
Posted by th...@apache.org.
DoFnSignature: Make TypeDescriptor-returning methods public
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/71fa7cdf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/71fa7cdf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/71fa7cdf
Branch: refs/heads/apex-runner
Commit: 71fa7cdf17e9719ae7fd606a5406ebe5821eb41e
Parents: 8336b24
Author: Kenneth Knowles <kl...@google.com>
Authored: Tue Nov 1 14:50:24 2016 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Thu Nov 3 21:32:53 2016 -0700
----------------------------------------------------------------------
.../beam/sdk/transforms/reflect/DoFnSignature.java | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71fa7cdf/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index 6b98805..431de02 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -341,7 +341,7 @@ public abstract class DoFnSignature {
/** Concrete type of the {@link RestrictionTracker} parameter, if present. */
@Nullable
- abstract TypeDescriptor<?> trackerT();
+ public abstract TypeDescriptor<?> trackerT();
/** Whether this {@link DoFn} returns a {@link ProcessContinuation} or void. */
public abstract boolean hasReturnValue();
@@ -461,7 +461,7 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Type of the returned restriction. */
- abstract TypeDescriptor<?> restrictionT();
+ public abstract TypeDescriptor<?> restrictionT();
static GetInitialRestrictionMethod create(Method targetMethod, TypeDescriptor<?> restrictionT) {
return new AutoValue_DoFnSignature_GetInitialRestrictionMethod(targetMethod, restrictionT);
@@ -476,7 +476,7 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Type of the restriction taken and returned. */
- abstract TypeDescriptor<?> restrictionT();
+ public abstract TypeDescriptor<?> restrictionT();
static SplitRestrictionMethod create(Method targetMethod, TypeDescriptor<?> restrictionT) {
return new AutoValue_DoFnSignature_SplitRestrictionMethod(targetMethod, restrictionT);
@@ -491,10 +491,10 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Type of the input restriction. */
- abstract TypeDescriptor<?> restrictionT();
+ public abstract TypeDescriptor<?> restrictionT();
/** Type of the returned {@link RestrictionTracker}. */
- abstract TypeDescriptor<?> trackerT();
+ public abstract TypeDescriptor<?> trackerT();
static NewTrackerMethod create(
Method targetMethod, TypeDescriptor<?> restrictionT, TypeDescriptor<?> trackerT) {
@@ -510,7 +510,7 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Type of the returned {@link Coder}. */
- abstract TypeDescriptor<?> coderT();
+ public abstract TypeDescriptor<?> coderT();
static GetRestrictionCoderMethod create(Method targetMethod, TypeDescriptor<?> coderT) {
return new AutoValue_DoFnSignature_GetRestrictionCoderMethod(targetMethod, coderT);
[03/50] incubator-beam git commit: [BEAM-854] Removes
ReifyTimestampsAndWindows
Posted by th...@apache.org.
[BEAM-854] Removes ReifyTimestampsAndWindows
It was used effectively in 2 places:
- GatherAllPanes (used in PAssert):
replaced with a similar custom DoFn.
- A couple of GBK-related transforms:
folded into GroupByKeyOnly.
Also makes GBKO produce globally-windowed KWIs.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e77d881a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e77d881a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e77d881a
Branch: refs/heads/apex-runner
Commit: e77d881aecc3f5fdbc407a252f9924c1226bcb4a
Parents: 529f266
Author: Eugene Kirpichov <ki...@google.com>
Authored: Fri Oct 28 10:40:17 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Wed Nov 2 10:18:59 2016 -0700
----------------------------------------------------------------------
.../core/GroupByKeyViaGroupByKeyOnly.java | 13 ++--
...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 2 -
.../beam/runners/direct/DirectGroupByKey.java | 42 ++++++-------
.../GroupAlsoByWindowEvaluatorFactory.java | 4 +-
.../direct/GroupByKeyOnlyEvaluatorFactory.java | 31 +++++-----
.../beam/runners/direct/ParDoEvaluator.java | 5 +-
.../direct/ParDoMultiEvaluatorHooks.java | 1 +
.../direct/ParDoSingleEvaluatorHooks.java | 1 +
.../direct/GroupByKeyEvaluatorFactoryTest.java | 29 ++++-----
.../GroupByKeyOnlyEvaluatorFactoryTest.java | 31 +++++-----
.../beam/runners/direct/ParDoEvaluatorTest.java | 5 +-
.../apache/beam/sdk/util/GatherAllPanes.java | 18 +++++-
.../sdk/util/ReifyTimestampAndWindowsDoFn.java | 41 -------------
.../sdk/util/ReifyTimestampsAndWindows.java | 63 --------------------
14 files changed, 94 insertions(+), 192 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java
----------------------------------------------------------------------
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java
index b521425..79d2252 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java
@@ -31,7 +31,6 @@ import org.apache.beam.sdk.transforms.OldDoFn;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.ReifyTimestampsAndWindows;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.util.WindowingStrategy;
@@ -84,15 +83,11 @@ public class GroupByKeyViaGroupByKeyOnly<K, V>
WindowingStrategy<?, ?> windowingStrategy = input.getWindowingStrategy();
return input
- // Make each input element's timestamp and assigned windows
- // explicit, in the value part.
- .apply(new ReifyTimestampsAndWindows<K, V>())
-
// Group by just the key.
// Combiner lifting will not happen regardless of the disallowCombinerLifting value.
// There will be no combiners right after the GroupByKeyOnly because of the two ParDos
// introduced in here.
- .apply(new GroupByKeyOnly<K, WindowedValue<V>>())
+ .apply(new GroupByKeyOnly<K, V>())
// Sort each key's values by timestamp. GroupAlsoByWindow requires
// its input to be sorted by timestamp.
@@ -112,12 +107,12 @@ public class GroupByKeyViaGroupByKeyOnly<K, V>
* or evaluate this class.
*/
public static class GroupByKeyOnly<K, V>
- extends PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> {
+ extends PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<WindowedValue<V>>>>> {
@SuppressWarnings({"rawtypes", "unchecked"})
@Override
- public PCollection<KV<K, Iterable<V>>> apply(PCollection<KV<K, V>> input) {
- return PCollection.<KV<K, Iterable<V>>>createPrimitiveOutputInternal(
+ public PCollection<KV<K, Iterable<WindowedValue<V>>>> apply(PCollection<KV<K, V>> input) {
+ return PCollection.createPrimitiveOutputInternal(
input.getPipeline(), input.getWindowingStrategy(), input.isBounded());
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
index b63e23b..680a971 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
@@ -24,7 +24,6 @@ import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.util.KeyedWorkItem;
import org.apache.beam.sdk.util.KeyedWorkItemCoder;
-import org.apache.beam.sdk.util.ReifyTimestampsAndWindows;
import org.apache.beam.sdk.util.WindowingStrategy;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
@@ -52,7 +51,6 @@ class DirectGBKIntoKeyedWorkItemsOverrideFactory<KeyT, InputT>
checkArgument(input.getCoder() instanceof KvCoder);
KvCoder<KeyT, InputT> kvCoder = (KvCoder<KeyT, InputT>) input.getCoder();
return input
- .apply(new ReifyTimestampsAndWindows<KeyT, InputT>())
// TODO: Perhaps windowing strategy should instead be set by ReifyTAW, or by DGBKO
.setWindowingStrategyInternal(WindowingStrategy.globalDefault())
.apply(new DirectGroupByKey.DirectGroupByKeyOnly<KeyT, InputT>())
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
index 14103a6..219314a 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
@@ -27,8 +27,6 @@ import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.util.KeyedWorkItem;
import org.apache.beam.sdk.util.KeyedWorkItemCoder;
-import org.apache.beam.sdk.util.ReifyTimestampsAndWindows;
-import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowingStrategy;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
@@ -55,13 +53,13 @@ class DirectGroupByKey<K, V>
// merging windows as needed, using the windows assigned to the
// key/value input elements and the window merge operation of the
// window function associated with the input PCollection.
- WindowingStrategy<?, ?> windowingStrategy = input.getWindowingStrategy();
+ WindowingStrategy<?, ?> inputWindowingStrategy = input.getWindowingStrategy();
+ // Update the windowing strategy as appropriate.
+ WindowingStrategy<?, ?> outputWindowingStrategy =
+ original.updateWindowingStrategy(inputWindowingStrategy);
// By default, implement GroupByKey via a series of lower-level operations.
return input
- // Make each input element's timestamp and assigned windows
- // explicit, in the value part.
- .apply(new ReifyTimestampsAndWindows<K, V>())
.apply(new DirectGroupByKeyOnly<K, V>())
.setCoder(
KeyedWorkItemCoder.of(
@@ -70,20 +68,20 @@ class DirectGroupByKey<K, V>
input.getWindowingStrategy().getWindowFn().windowCoder()))
// Group each key's values by window, merging windows as needed.
- .apply("GroupAlsoByWindow", new DirectGroupAlsoByWindow<K, V>(windowingStrategy))
+ .apply(
+ "GroupAlsoByWindow",
+ new DirectGroupAlsoByWindow<K, V>(inputWindowingStrategy, outputWindowingStrategy))
- // And update the windowing strategy as appropriate.
- .setWindowingStrategyInternal(original.updateWindowingStrategy(windowingStrategy))
.setCoder(
KvCoder.of(inputCoder.getKeyCoder(), IterableCoder.of(inputCoder.getValueCoder())));
}
static final class DirectGroupByKeyOnly<K, V>
- extends PTransform<PCollection<KV<K, WindowedValue<V>>>, PCollection<KeyedWorkItem<K, V>>> {
+ extends PTransform<PCollection<KV<K, V>>, PCollection<KeyedWorkItem<K, V>>> {
@Override
- public PCollection<KeyedWorkItem<K, V>> apply(PCollection<KV<K, WindowedValue<V>>> input) {
- return PCollection.<KeyedWorkItem<K, V>>createPrimitiveOutputInternal(
- input.getPipeline(), input.getWindowingStrategy(), input.isBounded());
+ public PCollection<KeyedWorkItem<K, V>> apply(PCollection<KV<K, V>> input) {
+ return PCollection.createPrimitiveOutputInternal(
+ input.getPipeline(), WindowingStrategy.globalDefault(), input.isBounded());
}
DirectGroupByKeyOnly() {}
@@ -92,14 +90,18 @@ class DirectGroupByKey<K, V>
static final class DirectGroupAlsoByWindow<K, V>
extends PTransform<PCollection<KeyedWorkItem<K, V>>, PCollection<KV<K, Iterable<V>>>> {
- private final WindowingStrategy<?, ?> windowingStrategy;
+ private final WindowingStrategy<?, ?> inputWindowingStrategy;
+ private final WindowingStrategy<?, ?> outputWindowingStrategy;
- public DirectGroupAlsoByWindow(WindowingStrategy<?, ?> windowingStrategy) {
- this.windowingStrategy = windowingStrategy;
+ public DirectGroupAlsoByWindow(
+ WindowingStrategy<?, ?> inputWindowingStrategy,
+ WindowingStrategy<?, ?> outputWindowingStrategy) {
+ this.inputWindowingStrategy = inputWindowingStrategy;
+ this.outputWindowingStrategy = outputWindowingStrategy;
}
- public WindowingStrategy<?, ?> getWindowingStrategy() {
- return windowingStrategy;
+ public WindowingStrategy<?, ?> getInputWindowingStrategy() {
+ return inputWindowingStrategy;
}
private KeyedWorkItemCoder<K, V> getKeyedWorkItemCoder(Coder<KeyedWorkItem<K, V>> inputCoder) {
@@ -125,8 +127,8 @@ class DirectGroupByKey<K, V>
@Override
public PCollection<KV<K, Iterable<V>>> apply(PCollection<KeyedWorkItem<K, V>> input) {
- return PCollection.<KV<K, Iterable<V>>>createPrimitiveOutputInternal(
- input.getPipeline(), input.getWindowingStrategy(), input.isBounded());
+ return PCollection.createPrimitiveOutputInternal(
+ input.getPipeline(), outputWindowingStrategy, input.isBounded());
}
}
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
index 4115bb7..37cc319 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
@@ -103,7 +103,8 @@ class GroupAlsoByWindowEvaluatorFactory implements TransformEvaluatorFactory {
@SuppressWarnings("unchecked")
WindowingStrategy<?, BoundedWindow> windowingStrategy =
- (WindowingStrategy<?, BoundedWindow>) application.getTransform().getWindowingStrategy();
+ (WindowingStrategy<?, BoundedWindow>)
+ application.getTransform().getInputWindowingStrategy();
DirectStepContext stepContext =
evaluationContext
@@ -125,6 +126,7 @@ class GroupAlsoByWindowEvaluatorFactory implements TransformEvaluatorFactory {
evaluationContext,
stepContext,
application,
+ windowingStrategy,
gabwDoFn,
Collections.<PCollectionView<?>>emptyList(),
MAIN_OUTPUT_TAG,
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java
index 2ead782..0fa7ebd 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java
@@ -67,14 +67,14 @@ class GroupByKeyOnlyEvaluatorFactory implements TransformEvaluatorFactory {
@Override
public void cleanup() {}
- private <K, V> TransformEvaluator<KV<K, WindowedValue<V>>> createEvaluator(
+ private <K, V> TransformEvaluator<KV<K, V>> createEvaluator(
final AppliedPTransform<
- PCollection<KV<K, WindowedValue<V>>>,
+ PCollection<KV<K, V>>,
PCollection<KeyedWorkItem<K, V>>,
DirectGroupByKeyOnly<K, V>>
application,
- final CommittedBundle<KV<K, WindowedValue<V>>> inputBundle) {
- return new GroupByKeyOnlyEvaluator<>(evaluationContext, inputBundle, application);
+ final CommittedBundle<KV<K, V>> inputBundle) {
+ return new GroupByKeyOnlyEvaluator<>(evaluationContext, application);
}
/**
@@ -84,12 +84,11 @@ class GroupByKeyOnlyEvaluatorFactory implements TransformEvaluatorFactory {
* @see GroupByKeyViaGroupByKeyOnly
*/
private static class GroupByKeyOnlyEvaluator<K, V>
- implements TransformEvaluator<KV<K, WindowedValue<V>>> {
+ implements TransformEvaluator<KV<K, V>> {
private final EvaluationContext evaluationContext;
- private final CommittedBundle<KV<K, WindowedValue<V>>> inputBundle;
private final AppliedPTransform<
- PCollection<KV<K, WindowedValue<V>>>,
+ PCollection<KV<K, V>>,
PCollection<KeyedWorkItem<K, V>>,
DirectGroupByKeyOnly<K, V>> application;
private final Coder<K> keyCoder;
@@ -97,19 +96,17 @@ class GroupByKeyOnlyEvaluatorFactory implements TransformEvaluatorFactory {
public GroupByKeyOnlyEvaluator(
EvaluationContext evaluationContext,
- CommittedBundle<KV<K, WindowedValue<V>>> inputBundle,
AppliedPTransform<
- PCollection<KV<K, WindowedValue<V>>>,
- PCollection<KeyedWorkItem<K, V>>,
+ PCollection<KV<K, V>>,
+ PCollection<KeyedWorkItem<K, V>>,
DirectGroupByKeyOnly<K, V>> application) {
this.evaluationContext = evaluationContext;
- this.inputBundle = inputBundle;
this.application = application;
this.keyCoder = getKeyCoder(application.getInput().getCoder());
this.groupingMap = new HashMap<>();
}
- private Coder<K> getKeyCoder(Coder<KV<K, WindowedValue<V>>> coder) {
+ private Coder<K> getKeyCoder(Coder<KV<K, V>> coder) {
checkState(
coder instanceof KvCoder,
"%s requires a coder of class %s."
@@ -118,13 +115,13 @@ class GroupByKeyOnlyEvaluatorFactory implements TransformEvaluatorFactory {
getClass().getSimpleName(),
KvCoder.class.getSimpleName());
@SuppressWarnings("unchecked")
- Coder<K> keyCoder = ((KvCoder<K, WindowedValue<V>>) coder).getKeyCoder();
+ Coder<K> keyCoder = ((KvCoder<K, V>) coder).getKeyCoder();
return keyCoder;
}
@Override
- public void processElement(WindowedValue<KV<K, WindowedValue<V>>> element) {
- KV<K, WindowedValue<V>> kv = element.getValue();
+ public void processElement(WindowedValue<KV<K, V>> element) {
+ KV<K, V> kv = element.getValue();
K key = kv.getKey();
byte[] encodedKey;
try {
@@ -139,10 +136,10 @@ class GroupByKeyOnlyEvaluatorFactory implements TransformEvaluatorFactory {
GroupingKey<K> groupingKey = new GroupingKey<>(key, encodedKey);
List<WindowedValue<V>> values = groupingMap.get(groupingKey);
if (values == null) {
- values = new ArrayList<WindowedValue<V>>();
+ values = new ArrayList<>();
groupingMap.put(groupingKey, values);
}
- values.add(kv.getValue());
+ values.add(element.withValue(kv.getValue()));
}
@Override
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
index 5913379..6f91319 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
@@ -31,9 +31,11 @@ import org.apache.beam.runners.core.PushbackSideInputDoFnRunner;
import org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext;
import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
import org.apache.beam.sdk.transforms.AppliedPTransform;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.ReadyCheckingSideInputReader;
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowingStrategy;
import org.apache.beam.sdk.util.state.CopyOnAccessInMemoryStateInternals;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
@@ -44,6 +46,7 @@ class ParDoEvaluator<InputT, OutputT> implements TransformEvaluator<InputT> {
EvaluationContext evaluationContext,
DirectStepContext stepContext,
AppliedPTransform<PCollection<InputT>, ?, ?> application,
+ WindowingStrategy<?, ? extends BoundedWindow> windowingStrategy,
Serializable fn, // may be OldDoFn or DoFn
List<PCollectionView<?>> sideInputs,
TupleTag<OutputT> mainOutputTag,
@@ -70,7 +73,7 @@ class ParDoEvaluator<InputT, OutputT> implements TransformEvaluator<InputT> {
sideOutputTags,
stepContext,
aggregatorChanges,
- application.getInput().getWindowingStrategy());
+ windowingStrategy);
PushbackSideInputDoFnRunner<InputT, OutputT> runner =
PushbackSideInputDoFnRunner.create(underlying, sideInputs, sideInputReader);
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiEvaluatorHooks.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiEvaluatorHooks.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiEvaluatorHooks.java
index a566154..f30f209 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiEvaluatorHooks.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiEvaluatorHooks.java
@@ -45,6 +45,7 @@ class ParDoMultiEvaluatorHooks<InputT, OutputT>
evaluationContext,
stepContext,
application,
+ application.getInput().getWindowingStrategy(),
fnLocal,
transform.getSideInputs(),
transform.getMainOutputTag(),
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorHooks.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorHooks.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorHooks.java
index b554f41..6d284c2 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorHooks.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorHooks.java
@@ -48,6 +48,7 @@ class ParDoSingleEvaluatorHooks<InputT, OutputT>
evaluationContext,
stepContext,
application,
+ application.getInput().getWindowingStrategy(),
fnLocal,
transform.getSideInputs(),
mainOutputTag,
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java
index 49d7d90..a726817 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java
@@ -35,7 +35,6 @@ import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.util.KeyedWorkItem;
import org.apache.beam.sdk.util.KeyedWorkItems;
-import org.apache.beam.sdk.util.ReifyTimestampsAndWindows;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
@@ -64,13 +63,11 @@ public class GroupByKeyEvaluatorFactoryTest {
KV<String, Integer> firstBaz = KV.of("baz", Integer.MAX_VALUE);
PCollection<KV<String, Integer>> values =
p.apply(Create.of(firstFoo, firstBar, secondFoo, firstBaz, secondBar, thirdFoo));
- PCollection<KV<String, WindowedValue<Integer>>> kvs =
- values.apply(new ReifyTimestampsAndWindows<String, Integer>());
PCollection<KeyedWorkItem<String, Integer>> groupedKvs =
- kvs.apply(new DirectGroupByKeyOnly<String, Integer>());
+ values.apply(new DirectGroupByKeyOnly<String, Integer>());
- CommittedBundle<KV<String, WindowedValue<Integer>>> inputBundle =
- bundleFactory.createBundle(kvs).commit(Instant.now());
+ CommittedBundle<KV<String, Integer>> inputBundle =
+ bundleFactory.createBundle(values).commit(Instant.now());
EvaluationContext evaluationContext = mock(EvaluationContext.class);
StructuralKey<String> fooKey = StructuralKey.of("foo", StringUtf8Coder.of());
UncommittedBundle<KeyedWorkItem<String, Integer>> fooBundle =
@@ -97,17 +94,17 @@ public class GroupByKeyEvaluatorFactoryTest {
// The input to a GroupByKey is assumed to be a KvCoder
@SuppressWarnings("unchecked")
Coder<String> keyCoder =
- ((KvCoder<String, WindowedValue<Integer>>) kvs.getCoder()).getKeyCoder();
- TransformEvaluator<KV<String, WindowedValue<Integer>>> evaluator =
+ ((KvCoder<String, Integer>) values.getCoder()).getKeyCoder();
+ TransformEvaluator<KV<String, Integer>> evaluator =
new GroupByKeyOnlyEvaluatorFactory(evaluationContext)
.forApplication(groupedKvs.getProducingTransformInternal(), inputBundle);
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(firstFoo)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(secondFoo)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(thirdFoo)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(firstBar)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(secondBar)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(firstBaz)));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(firstFoo));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(secondFoo));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(thirdFoo));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(firstBar));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(secondBar));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(firstBaz));
evaluator.finishBundle();
@@ -142,10 +139,6 @@ public class GroupByKeyEvaluatorFactoryTest {
keyCoder)));
}
- private <K, V> KV<K, WindowedValue<V>> gwValue(KV<K, V> kv) {
- return KV.of(kv.getKey(), WindowedValue.valueInGlobalWindow(kv.getValue()));
- }
-
private static class KeyedWorkItemMatcher<K, V>
extends BaseMatcher<WindowedValue<KeyedWorkItem<K, V>>> {
private final KeyedWorkItem<K, V> myWorkItem;
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java
index 3b9dc39..3e5af14 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java
@@ -35,7 +35,6 @@ import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.util.KeyedWorkItem;
import org.apache.beam.sdk.util.KeyedWorkItems;
-import org.apache.beam.sdk.util.ReifyTimestampsAndWindows;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
@@ -64,13 +63,11 @@ public class GroupByKeyOnlyEvaluatorFactoryTest {
KV<String, Integer> firstBaz = KV.of("baz", Integer.MAX_VALUE);
PCollection<KV<String, Integer>> values =
p.apply(Create.of(firstFoo, firstBar, secondFoo, firstBaz, secondBar, thirdFoo));
- PCollection<KV<String, WindowedValue<Integer>>> kvs =
- values.apply(new ReifyTimestampsAndWindows<String, Integer>());
PCollection<KeyedWorkItem<String, Integer>> groupedKvs =
- kvs.apply(new DirectGroupByKeyOnly<String, Integer>());
+ values.apply(new DirectGroupByKeyOnly<String, Integer>());
- CommittedBundle<KV<String, WindowedValue<Integer>>> inputBundle =
- bundleFactory.createBundle(kvs).commit(Instant.now());
+ CommittedBundle<KV<String, Integer>> inputBundle =
+ bundleFactory.createBundle(values).commit(Instant.now());
EvaluationContext evaluationContext = mock(EvaluationContext.class);
StructuralKey<String> fooKey = StructuralKey.of("foo", StringUtf8Coder.of());
@@ -90,25 +87,25 @@ public class GroupByKeyOnlyEvaluatorFactoryTest {
// The input to a GroupByKey is assumed to be a KvCoder
@SuppressWarnings("unchecked")
Coder<String> keyCoder =
- ((KvCoder<String, WindowedValue<Integer>>) kvs.getCoder()).getKeyCoder();
- TransformEvaluator<KV<String, WindowedValue<Integer>>> evaluator =
+ ((KvCoder<String, Integer>) values.getCoder()).getKeyCoder();
+ TransformEvaluator<KV<String, Integer>> evaluator =
new GroupByKeyOnlyEvaluatorFactory(evaluationContext)
.forApplication(
groupedKvs.getProducingTransformInternal(), inputBundle);
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(firstFoo)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(secondFoo)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(thirdFoo)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(firstBar)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(secondBar)));
- evaluator.processElement(WindowedValue.valueInEmptyWindows(gwValue(firstBaz)));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(firstFoo));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(secondFoo));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(thirdFoo));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(firstBar));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(secondBar));
+ evaluator.processElement(WindowedValue.valueInGlobalWindow(firstBaz));
evaluator.finishBundle();
assertThat(
fooBundle.commit(Instant.now()).getElements(),
contains(
- new KeyedWorkItemMatcher<String, Integer>(
+ new KeyedWorkItemMatcher<>(
KeyedWorkItems.elementsWorkItem(
"foo",
ImmutableSet.of(
@@ -119,7 +116,7 @@ public class GroupByKeyOnlyEvaluatorFactoryTest {
assertThat(
barBundle.commit(Instant.now()).getElements(),
contains(
- new KeyedWorkItemMatcher<String, Integer>(
+ new KeyedWorkItemMatcher<>(
KeyedWorkItems.elementsWorkItem(
"bar",
ImmutableSet.of(
@@ -129,7 +126,7 @@ public class GroupByKeyOnlyEvaluatorFactoryTest {
assertThat(
bazBundle.commit(Instant.now()).getElements(),
contains(
- new KeyedWorkItemMatcher<String, Integer>(
+ new KeyedWorkItemMatcher<>(
KeyedWorkItems.elementsWorkItem(
"baz",
ImmutableSet.of(WindowedValue.valueInGlobalWindow(Integer.MAX_VALUE))),
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
index 8254413..eab92f4 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
@@ -152,10 +152,13 @@ public class ParDoEvaluatorTest {
when(evaluationContext.getAggregatorContainer()).thenReturn(container);
when(evaluationContext.getAggregatorMutator()).thenReturn(mutator);
+ AppliedPTransform<PCollection<Integer>, ?, ?> transform =
+ (AppliedPTransform<PCollection<Integer>, ?, ?>) output.getProducingTransformInternal();
return ParDoEvaluator.create(
evaluationContext,
stepContext,
- (AppliedPTransform<PCollection<Integer>, ?, ?>) output.getProducingTransformInternal(),
+ transform,
+ transform.getInput().getWindowingStrategy(),
fn,
ImmutableList.<PCollectionView<?>>of(singletonView),
mainOutputTag,
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GatherAllPanes.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GatherAllPanes.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GatherAllPanes.java
index a2a6e17..52a2ba8 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GatherAllPanes.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GatherAllPanes.java
@@ -17,10 +17,13 @@
*/
package org.apache.beam.sdk.util;
+import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.Never;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -55,8 +58,12 @@ public class GatherAllPanes<T>
WindowFn<?, ?> originalWindowFn = input.getWindowingStrategy().getWindowFn();
return input
- .apply(WithKeys.<Integer, T>of(0).withKeyType(new TypeDescriptor<Integer>() {}))
- .apply(new ReifyTimestampsAndWindows<Integer, T>())
+ .apply(ParDo.of(new ReifyTimestampsAndWindowsFn<T>()))
+ .setCoder(
+ WindowedValue.FullWindowedValueCoder.of(
+ input.getCoder(), input.getWindowingStrategy().getWindowFn().windowCoder()))
+ .apply(
+ WithKeys.<Integer, WindowedValue<T>>of(0).withKeyType(new TypeDescriptor<Integer>() {}))
.apply(
Window.into(
new IdentityWindowFn<KV<Integer, WindowedValue<T>>>(
@@ -69,4 +76,11 @@ public class GatherAllPanes<T>
.apply(Values.<Iterable<WindowedValue<T>>>create())
.setWindowingStrategyInternal(input.getWindowingStrategy());
}
+
+ private static class ReifyTimestampsAndWindowsFn<T> extends DoFn<T, WindowedValue<T>> {
+ @DoFn.ProcessElement
+ public void processElement(ProcessContext c, BoundedWindow window) {
+ c.output(WindowedValue.of(c.element(), c.timestamp(), window, c.pane()));
+ }
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestampAndWindowsDoFn.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestampAndWindowsDoFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestampAndWindowsDoFn.java
deleted file mode 100644
index 6da4da0..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestampAndWindowsDoFn.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import org.apache.beam.sdk.transforms.OldDoFn;
-import org.apache.beam.sdk.transforms.OldDoFn.RequiresWindowAccess;
-import org.apache.beam.sdk.values.KV;
-
-/**
- * {@link OldDoFn} that makes timestamps and window assignments explicit in the value part of each
- * key/value pair.
- *
- * @param <K> the type of the keys of the input and output {@code PCollection}s
- * @param <V> the type of the values of the input {@code PCollection}
- */
-@SystemDoFnInternal
-public class ReifyTimestampAndWindowsDoFn<K, V> extends OldDoFn<KV<K, V>, KV<K, WindowedValue<V>>>
- implements RequiresWindowAccess {
- @Override
- public void processElement(ProcessContext c) throws Exception {
- KV<K, V> kv = c.element();
- K key = kv.getKey();
- V value = kv.getValue();
- c.output(KV.of(key, WindowedValue.of(value, c.timestamp(), c.window(), c.pane())));
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e77d881a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestampsAndWindows.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestampsAndWindows.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestampsAndWindows.java
deleted file mode 100644
index d129c8e..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestampsAndWindows.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
- * implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.util;
-
-import static com.google.common.base.Preconditions.checkArgument;
-
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-
-/**
- * Helper transform that makes timestamps and window assignments explicit in the value part of
- * each key/value pair.
- */
-public class ReifyTimestampsAndWindows<K, V>
- extends PTransform<PCollection<KV<K, V>>, PCollection<KV<K, WindowedValue<V>>>> {
-
- @Override
- public PCollection<KV<K, WindowedValue<V>>> apply(PCollection<KV<K, V>> input) {
-
- // The requirement to use a KvCoder *is* actually a model-level requirement, not specific
- // to this implementation of GBK. All runners need a way to get the key.
- checkArgument(
- input.getCoder() instanceof KvCoder,
- "%s requires its input to use a %s",
- GroupByKey.class.getSimpleName(),
- KvCoder.class.getSimpleName());
-
- @SuppressWarnings("unchecked")
- KvCoder<K, V> inputKvCoder = (KvCoder<K, V>) input.getCoder();
- Coder<K> keyCoder = inputKvCoder.getKeyCoder();
- Coder<V> inputValueCoder = inputKvCoder.getValueCoder();
- Coder<WindowedValue<V>> outputValueCoder =
- FullWindowedValueCoder.of(
- inputValueCoder, input.getWindowingStrategy().getWindowFn().windowCoder());
- Coder<KV<K, WindowedValue<V>>> outputKvCoder = KvCoder.of(keyCoder, outputValueCoder);
- return input
- .apply(ParDo.of(new ReifyTimestampAndWindowsDoFn<K, V>()))
- .setCoder(outputKvCoder);
- }
-}
[40/50] incubator-beam git commit: Fix javadoc throughout Beam
Posted by th...@apache.org.
Fix javadoc throughout Beam
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/717b431c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/717b431c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/717b431c
Branch: refs/heads/apex-runner
Commit: 717b431c1aeeeec4bad279944c2a1a406b3f08ee
Parents: aae65db
Author: Dan Halperin <dh...@google.com>
Authored: Fri Nov 4 18:28:53 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Sat Nov 5 01:30:40 2016 -0700
----------------------------------------------------------------------
.../beam/examples/complete/game/GameStats.java | 3 +-
.../examples/complete/game/LeaderBoard.java | 2 +-
.../beam/examples/complete/game/UserScore.java | 2 +-
.../flink/examples/streaming/AutoComplete.java | 8 ++---
.../flink/examples/streaming/JoinExamples.java | 12 +++----
.../examples/streaming/KafkaIOExamples.java | 2 +-
.../examples/streaming/WindowedWordCount.java | 8 ++---
.../runners/dataflow/DataflowPipelineJob.java | 1 -
.../runners/dataflow/internal/IsmFormat.java | 5 +--
.../runners/dataflow/util/MonitoringUtil.java | 1 -
.../runners/dataflow/util/RandomAccessData.java | 2 +-
.../java/org/apache/beam/sdk/io/AvroIO.java | 4 +--
.../org/apache/beam/sdk/io/BoundedSource.java | 11 +++---
.../apache/beam/sdk/io/CompressedSource.java | 2 --
.../org/apache/beam/sdk/io/FileBasedSink.java | 9 ++---
.../java/org/apache/beam/sdk/io/XmlSource.java | 2 +-
.../apache/beam/sdk/io/range/ByteKeyRange.java | 2 +-
.../apache/beam/sdk/options/ValueProvider.java | 2 +-
.../apache/beam/sdk/transforms/Aggregator.java | 12 +++----
.../apache/beam/sdk/transforms/CombineFns.java | 4 +--
.../org/apache/beam/sdk/transforms/DoFn.java | 22 ++++++------
.../org/apache/beam/sdk/transforms/ParDo.java | 2 +-
.../org/apache/beam/sdk/transforms/View.java | 2 +-
.../sdk/transforms/display/DisplayData.java | 5 ++-
.../beam/sdk/transforms/join/CoGroupByKey.java | 6 ++--
.../sdk/transforms/reflect/DoFnSignature.java | 2 +-
.../splittabledofn/RestrictionTracker.java | 2 +-
.../transforms/splittabledofn/package-info.java | 4 +--
.../beam/sdk/util/BaseExecutionContext.java | 2 +-
.../BufferedElementCountingOutputStream.java | 4 +--
.../apache/beam/sdk/util/ExecutionContext.java | 8 ++---
.../sdk/util/ExposedByteArrayInputStream.java | 5 +--
.../sdk/util/ExposedByteArrayOutputStream.java | 2 --
.../java/org/apache/beam/sdk/util/GcsUtil.java | 2 --
.../apache/beam/sdk/util/MovingFunction.java | 4 +--
.../org/apache/beam/sdk/util/PubsubClient.java | 32 +++---------------
.../org/apache/beam/sdk/util/StringUtils.java | 8 ++---
.../apache/beam/sdk/util/TimerInternals.java | 8 +++--
.../beam/sdk/util/state/StateNamespace.java | 16 ++++-----
.../java/org/apache/beam/sdk/values/PDone.java | 2 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 2 +-
.../apache/beam/sdk/io/kinesis/KinesisIO.java | 35 +++++++++-----------
42 files changed, 109 insertions(+), 160 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
----------------------------------------------------------------------
diff --git a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
index 5ebf892..2b5255f 100644
--- a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
+++ b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
@@ -100,7 +100,8 @@ public class GameStats extends LeaderBoard {
/**
* Filter out all but those users with a high clickrate, which we will consider as 'spammy' uesrs.
* We do this by finding the mean total score per user, then using that information as a side
- * input to filter out all but those user scores that are > (mean * SCORE_WEIGHT)
+ * input to filter out all but those user scores that are larger than
+ * {@code (mean * SCORE_WEIGHT)}.
*/
// [START DocInclude_AbuseDetect]
public static class CalculateSpammyUsers
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
----------------------------------------------------------------------
diff --git a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
index d5e3345..0a3dfb3 100644
--- a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
+++ b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
@@ -73,7 +73,7 @@ import org.joda.time.format.DateTimeFormatter;
* results, e.g. for 'team prizes'. We're now outputting window results as they're
* calculated, giving us much lower latency than with the previous batch examples.
*
- * <p>Run {@link injector.Injector} to generate pubsub data for this pipeline. The Injector
+ * <p>Run {@code injector.Injector} to generate pubsub data for this pipeline. The Injector
* documentation provides more detail on how to do this.
*
* <p>To execute this pipeline using the Dataflow service, specify the pipeline configuration
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
----------------------------------------------------------------------
diff --git a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
index 32c939f..f4dca32 100644
--- a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
+++ b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
@@ -68,7 +68,7 @@ import org.slf4j.LoggerFactory;
* where the BigQuery dataset you specify must already exist.
*
* <p>Optionally include the --input argument to specify a batch input file.
- * See the --input default value for example batch data file, or use {@link injector.Injector} to
+ * See the --input default value for example batch data file, or use {@code injector.Injector} to
* generate your own batch data.
*/
public class UserScore {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/AutoComplete.java
----------------------------------------------------------------------
diff --git a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/AutoComplete.java b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/AutoComplete.java
index 97ba232..1e0c3ac 100644
--- a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/AutoComplete.java
+++ b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/AutoComplete.java
@@ -53,11 +53,9 @@ import org.joda.time.Duration;
/**
* To run the example, first open a socket on a terminal by executing the command:
- * <li>
- * <li>
- * <code>nc -lk 9999</code>
- * </li>
- * </li>
+ * <ul>
+ * <li><code>nc -lk 9999</code>
+ * </ul>
* and then launch the example. Now whatever you type in the terminal is going to be
* the input to the program.
* */
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/JoinExamples.java
----------------------------------------------------------------------
diff --git a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/JoinExamples.java b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/JoinExamples.java
index 96638aa..8fefc9f 100644
--- a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/JoinExamples.java
+++ b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/JoinExamples.java
@@ -39,14 +39,10 @@ import org.joda.time.Duration;
/**
* To run the example, first open two sockets on two terminals by executing the commands:
- * <li>
- * <li>
- * <code>nc -lk 9999</code>, and
- * </li>
- * <li>
- * <code>nc -lk 9998</code>
- * </li>
- * </li>
+ * <ul>
+ * <li><code>nc -lk 9999</code>, and
+ * <li><code>nc -lk 9998</code>
+ * </ul>
* and then launch the example. Now whatever you type in the terminal is going to be
* the input to the program.
* */
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaIOExamples.java
----------------------------------------------------------------------
diff --git a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaIOExamples.java b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaIOExamples.java
index f0bf188..3c8a89b 100644
--- a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaIOExamples.java
+++ b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaIOExamples.java
@@ -170,7 +170,7 @@ public class KafkaIOExamples {
/**
* Serialiation/Deserialiation schema for Avro types.
- * @param <T>
+ * @param <T> the type being encoded
*/
static class AvroSerializationDeserializationSchema<T>
implements SerializationSchema<T>, DeserializationSchema<T> {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/WindowedWordCount.java
----------------------------------------------------------------------
diff --git a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/WindowedWordCount.java b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/WindowedWordCount.java
index 0e250b8..2246bdd 100644
--- a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/WindowedWordCount.java
+++ b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/WindowedWordCount.java
@@ -42,11 +42,9 @@ import org.slf4j.LoggerFactory;
/**
* To run the example, first open a socket on a terminal by executing the command:
- * <li>
- * <li>
- * <code>nc -lk 9999</code>
- * </li>
- * </li>
+ * <ul>
+ * <li><code>nc -lk 9999</code>
+ * </ul>
* and then launch the example. Now whatever you type in the terminal is going to be
* the input to the program.
* */
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
----------------------------------------------------------------------
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index 27006a4..a2b632f 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -199,7 +199,6 @@ public class DataflowPipelineJob implements PipelineResult {
* thread is interrupted.
* @throws IOException If there is a persistent problem getting job
* information.
- * @throws InterruptedException
*/
@Nullable
@VisibleForTesting
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java
----------------------------------------------------------------------
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java
index 903e7b4..2f83ffd 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java
@@ -167,11 +167,12 @@ public class IsmFormat {
}
}
- /** A {@link Coder} for {@link IsmRecord}s.
+ /**
+ * A {@link Coder} for {@link IsmRecord}s.
*
* <p>Note that this coder standalone will not produce an Ism file. This coder can be used
* to materialize a {@link PCollection} of {@link IsmRecord}s. Only when this coder
- * is combined with an {@link IsmSink} will one produce an Ism file.
+ * is combined with an {@code IsmSink} will one produce an Ism file.
*
* <p>The {@link IsmRecord} encoded format is:
* <ul>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
----------------------------------------------------------------------
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
index a6afccd..efb6d2b 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
@@ -149,7 +149,6 @@ public final class MonitoringUtil {
* @param startTimestampMs Return only those messages with a
* timestamp greater than this value.
* @return collection of messages
- * @throws IOException
*/
public ArrayList<JobMessage> getJobMessages(
String jobId, long startTimestampMs) throws IOException {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java
----------------------------------------------------------------------
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java
index 84c53ea..11eec19 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java
@@ -121,7 +121,7 @@ public class RandomAccessData {
* A {@link Comparator} that compares two byte arrays lexicographically. It compares
* values as a list of unsigned bytes. The first pair of values that follow any common prefix,
* or when one array is a prefix of the other, treats the shorter array as the lesser.
- * For example, [] < [0x01] < [0x01, 0x7F] < [0x01, 0x80] < [0x02] < POSITIVE INFINITY.
+ * For example, {@code [] < [0x01] < [0x01, 0x7F] < [0x01, 0x80] < [0x02] < POSITIVE INFINITY}.
*
* <p>Note that a token type of positive infinity is supported and is greater than
* all other {@link RandomAccessData}.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
index 6deca7f..bd0c655 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
@@ -117,10 +117,10 @@ import org.apache.beam.sdk.values.PDone;
* } </pre>
*
* <p>By default, {@link AvroIO.Write} produces output files that are compressed using the
- * {@link org.apache.avro.file.DeflateCodec CodecFactory.deflateCodec(6)}. This default can
+ * {@link org.apache.avro.file.Codec CodecFactory.deflateCodec(6)}. This default can
* be changed or overridden using {@link AvroIO.Write#withCodec}.
*
- * <p><h3>Permissions</h3>
+ * <h3>Permissions</h3>
* Permission requirements depend on the {@link PipelineRunner} that is used to execute the
* Dataflow job. Please refer to the documentation of corresponding {@link PipelineRunner}s for
* more details.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java
index cd4d7db..b3dd138 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java
@@ -50,9 +50,6 @@ import org.joda.time.Instant;
* </li>
* </ul>
*
- * <p>To use this class for supporting your custom input type, derive your class
- * class from it, and override the abstract methods. For an example, see {@link DatastoreIO}.
- *
* @param <T> Type of records read by the source.
*/
public abstract class BoundedSource<T> extends Source<T> {
@@ -135,7 +132,7 @@ public abstract class BoundedSource<T> extends Source<T> {
*
* <p>By default, returns null to indicate that this cannot be estimated.
*
- * <h5>Thread safety</h5>
+ * <h3>Thread safety</h3>
* If {@link #splitAtFraction} is implemented, this method can be called concurrently to other
* methods (including itself), and it is therefore critical for it to be implemented
* in a thread-safe way.
@@ -345,7 +342,7 @@ public abstract class BoundedSource<T> extends Source<T> {
*
* <p>Returns a {@code BoundedSource} representing the remainder.
*
- * <h5>Detailed description</h5>
+ * <h3>Detailed description</h3>
* Assuming the following sequence of calls:
* <pre>{@code
* BoundedSource<T> initial = reader.getCurrentSource();
@@ -371,11 +368,11 @@ public abstract class BoundedSource<T> extends Source<T> {
* corresponding to the given fraction. In this case, the method MUST have no effect
* (the reader must behave as if the method hadn't been called at all).
*
- * <h5>Statefulness</h5>
+ * <h3>Statefulness</h3>
* Since this method (if successful) affects the reader's source, in subsequent invocations
* "fraction" should be interpreted relative to the new current source.
*
- * <h5>Thread safety and blocking</h5>
+ * <h3>Thread safety and blocking</h3>
* This method will be called concurrently to other methods (however there will not be multiple
* concurrent invocations of this method itself), and it is critical for it to be implemented
* in a thread-safe way (otherwise data loss is possible).
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java
index f33b9bd..af0c8130 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java
@@ -74,7 +74,6 @@ public class CompressedSource<T> extends FileBasedSource<T> {
public interface DecompressingChannelFactory extends Serializable {
/**
* Given a channel, create a channel that decompresses the content read from the channel.
- * @throws IOException
*/
ReadableByteChannel createDecompressingChannel(ReadableByteChannel channel)
throws IOException;
@@ -88,7 +87,6 @@ public class CompressedSource<T> extends FileBasedSource<T> {
extends DecompressingChannelFactory {
/**
* Given a channel, create a channel that decompresses the content read from the channel.
- * @throws IOException
*/
ReadableByteChannel createDecompressingChannel(String fileName, ReadableByteChannel channel)
throws IOException;
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index 3d0fe04..101ff61 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -133,7 +133,8 @@ public abstract class FileBasedSink<T> extends Sink<T> {
/**
* The {@link WritableByteChannelFactory} that is used to wrap the raw data output to the
- * underlying channel. The default is to not compress the output using {@link #UNCOMPRESSED}.
+ * underlying channel. The default is to not compress the output using
+ * {@link CompressionType#UNCOMPRESSED}.
*/
protected final WritableByteChannelFactory writableByteChannelFactory;
@@ -206,11 +207,6 @@ public abstract class FileBasedSink<T> extends Sink<T> {
return baseOutputFilename;
}
- /**
- * Perform pipeline-construction-time validation. The default implementation is a no-op.
- * Subclasses should override to ensure the sink is valid and can be written to. It is recommended
- * to use {@link Preconditions#checkState(boolean)} in the implementation of this method.
- */
@Override
public void validate(PipelineOptions options) {}
@@ -802,7 +798,6 @@ public abstract class FileBasedSink<T> extends Sink<T> {
/**
* @param channel the {@link WritableByteChannel} to wrap
* @return the {@link WritableByteChannel} to be used during output
- * @throws IOException
*/
WritableByteChannel create(WritableByteChannel channel) throws IOException;
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/io/XmlSource.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/XmlSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/XmlSource.java
index aaee75a..1458f74 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/XmlSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/XmlSource.java
@@ -108,7 +108,7 @@ import org.codehaus.stax2.XMLInputFactory2;
* <p>These dependencies have been declared as optional in the sdks/java/core/pom.xml file of
* Apache Beam.
*
- * <p><h3>Permissions</h3>
+ * <h3>Permissions</h3>
* Permission requirements depend on the
* {@link org.apache.beam.sdk.runners.PipelineRunner PipelineRunner} that is
* used to execute the Beam pipeline. Please refer to the documentation of corresponding
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java
index 3063441..e30f8af 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java
@@ -133,7 +133,7 @@ public final class ByteKeyRange implements Serializable {
* Specifically, if this range is unsplittable (e.g., because the start and end keys are equal
* up to padding by zero bytes), the list returned will only contain the start and end key.
*
- * @throws IllegalArgumentException if the specified number of splits is < 1
+ * @throws IllegalArgumentException if the specified number of splits is less than 1
* @see ByteKeyRange the ByteKeyRange class Javadoc for more information about split semantics.
*/
public List<ByteKey> split(int numSplits) {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
index 4f0280b..188b371 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
@@ -133,7 +133,7 @@ public interface ValueProvider<T> {
* construction time.
*
* <p>To enforce this contract, if there is no default, users must only call
- * {@link #get()} at execution time (after a call to {@link Pipeline#run}),
+ * {@link #get()} at execution time (after a call to {@link org.apache.beam.sdk.Pipeline#run}),
* which will provide the value of {@code optionsMap}.
*/
class RuntimeValueProvider<T> implements ValueProvider<T>, Serializable {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Aggregator.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Aggregator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Aggregator.java
index 427ecfc..43f53a8 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Aggregator.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Aggregator.java
@@ -29,13 +29,11 @@ import org.apache.beam.sdk.util.ExecutionContext;
* typically from the {@link DoFn} constructor. Elements can be added to the
* {@code Aggregator} by calling {@link Aggregator#addValue}.
*
- * <p>Aggregators are visible in the monitoring UI, when the pipeline is run using
- * {@link DataflowRunner} along with their current value.
- * Aggregators may not become visible until the system begins executing the ParDo transform
- * that created them and/or their initial value is changed.
+ * <p>It is runner-dependent whether aggregators are accessible during pipeline execution or only
+ * after jobs have completed.
*
* <p>Example:
- * <pre> {@code
+ * <pre>{@code
* class MyDoFn extends DoFn<String, String> {
* private Aggregator<Integer, Integer> myAggregator;
*
@@ -43,12 +41,12 @@ import org.apache.beam.sdk.util.ExecutionContext;
* myAggregator = createAggregator("myAggregator", new Sum.SumIntegerFn());
* }
*
- * @ProcessElement
+ * {@literal @}ProcessElement
* public void processElement(ProcessContext c) {
* myAggregator.addValue(1);
* }
* }
- * } </pre>
+ * }</pre>
*
* @param <InputT> the type of input values
* @param <OutputT> the type of output values
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
index 1b3e525..e4e1c50 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
@@ -106,7 +106,7 @@ public class CombineFns {
* <p>The same {@link TupleTag} cannot be used in a composition multiple times.
*
* <p>Example:
- * <pre><{@code
+ * <pre>{@code
* PCollection<Integer> globalLatencies = ...;
*
* TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>();
@@ -114,7 +114,7 @@ public class CombineFns {
*
* SimpleFunction<Integer, Integer> identityFn =
* new SimpleFunction<Integer, Integer>() {
- * @Override
+ * {@literal @}Override
* public Integer apply(Integer input) {
* return input;
* }};
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index 018877f..2b3962e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -76,15 +76,15 @@ import org.joda.time.Instant;
*
* <p>Example usage:
*
- * <pre> {@code
+ * <pre>{@code
* PCollection<String> lines = ... ;
* PCollection<String> words =
* lines.apply(ParDo.of(new DoFn<String, String>() {
- * @ProcessElement
+ * {@literal @}ProcessElement
* public void processElement(ProcessContext c, BoundedWindow window) {
*
* }}));
- * } </pre>
+ * }</pre>
*
* @param <InputT> the type of the (main) input elements
* @param <OutputT> the type of the (main) output elements
@@ -436,14 +436,14 @@ public abstract class DoFn<InputT, OutputT> implements Serializable, HasDisplayD
*
* <pre>{@code
* new DoFn<KV<Key, Foo>, Baz>() {
- * @StateId("my-state-id")
+ * {@literal @}StateId("my-state-id")
* private final StateSpec<K, ValueState<MyState>> myStateSpec =
* StateSpecs.value(new MyStateCoder());
*
- * @ProcessElement
+ * {@literal @}ProcessElement
* public void processElement(
* ProcessContext c,
- * @StateId("my-state-id") ValueState<MyState> myState) {
+ * {@literal @}StateId("my-state-id") ValueState<MyState> myState) {
* myState.read();
* myState.write(...);
* }
@@ -480,17 +480,17 @@ public abstract class DoFn<InputT, OutputT> implements Serializable, HasDisplayD
*
* <pre>{@code
* new DoFn<KV<Key, Foo>, Baz>() {
- * @TimerId("my-timer-id")
+ * {@literal @}TimerId("my-timer-id")
* private final TimerSpec myTimer = TimerSpecs.timerForDomain(TimeDomain.EVENT_TIME);
*
- * @ProcessElement
+ * {@literal @}ProcessElement
* public void processElement(
* ProcessContext c,
- * @TimerId("my-timer-id") Timer myTimer) {
+ * {@literal @}TimerId("my-timer-id") Timer myTimer) {
* myTimer.setForNowPlus(Duration.standardSeconds(...));
* }
*
- * @OnTimer("my-timer-id")
+ * {@literal @}OnTimer("my-timer-id")
* public void onMyTimer() {
* ...
* }
@@ -578,7 +578,7 @@ public abstract class DoFn<InputT, OutputT> implements Serializable, HasDisplayD
* <ul>
* <li>Its first argument must be a {@link DoFn.ProcessContext}.
* <li>If one of its arguments is a subtype of {@link RestrictionTracker}, then it is a <a
- * href="https://s.apache.org/splittable-do-fn>splittable</a> {@link DoFn} subject to the
+ * href="https://s.apache.org/splittable-do-fn">splittable</a> {@link DoFn} subject to the
* separate requirements described below. Items below are assuming this is not a splittable
* {@link DoFn}.
* <li>If one of its arguments is {@link BoundedWindow}, this argument corresponds to the window
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index a3a306a..0684a5c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -96,7 +96,7 @@ import org.apache.beam.sdk.values.TypedPValue;
*
* <p>For example:
*
- * <pre><{@code
+ * <pre>{@code
* PCollection<String> lines = ...;
* PCollection<String> words =
* lines.apply(ParDo.of(new DoFn<String, String>() {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
index d750efc..5fafc0a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
@@ -213,7 +213,7 @@ public class View {
/**
* Returns a {@link View.AsMultimap} transform that takes a
- * {@link PCollection PCollection<KV<K, V>>}
+ * {@link PCollection PCollection<KV<K, V>>}
* as input and produces a {@link PCollectionView} mapping
* each window to its contents as a {@link Map Map<K, Iterable<V>>}
* for use as a side input.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
index 46cf94a..1a2de97 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
@@ -87,9 +87,8 @@ public class DisplayData implements Serializable {
*
* <p>Use this method if the type of metadata is not known at compile time. For example:
*
- * <pre>
- * {@code
- * @Override
+ * <pre>{@code
+ * {@literal @}Override
* public void populateDisplayData(DisplayData.Builder builder) {
* Optional<DisplayData.Type> type = DisplayData.inferType(foo);
* if (type.isPresent()) {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGroupByKey.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGroupByKey.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGroupByKey.java
index 9546b29..b760e2c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGroupByKey.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGroupByKey.java
@@ -43,7 +43,7 @@ import org.apache.beam.sdk.values.PCollectionList;
* <p>Example of performing a {@link CoGroupByKey} followed by a
* {@link ParDo} that consumes
* the results:
- * <pre> {@code
+ * <pre>{@code
* PCollection<KV<K, V1>> pt1 = ...;
* PCollection<KV<K, V2>> pt2 = ...;
*
@@ -57,7 +57,7 @@ import org.apache.beam.sdk.values.PCollectionList;
* PCollection<T> finalResultCollection =
* coGbkResultCollection.apply(ParDo.of(
* new DoFn<KV<K, CoGbkResult>, T>() {
- * @Override
+ * {@literal @}ProcessElement
* public void processElement(ProcessContext c) {
* KV<K, CoGbkResult> e = c.element();
* Iterable<V1> pt1Vals = e.getValue().getAll(t1);
@@ -66,7 +66,7 @@ import org.apache.beam.sdk.values.PCollectionList;
* c.output(...some T...);
* }
* }));
- * } </pre>
+ * }</pre>
*
* @param <K> the type of the keys in the input and output
* {@code PCollection}s
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index 7087efa..11f6aa7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -415,7 +415,7 @@ public abstract class DoFnSignature {
/**
* Describes a timer declaration; a field of type {@link TimerSpec} annotated with
- * {@DoFn.TimerId}.
+ * {@link DoFn.TimerId}.
*/
@AutoValue
public abstract static class TimerDeclaration {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/RestrictionTracker.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/RestrictionTracker.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/RestrictionTracker.java
index 6b249ee..268766b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/RestrictionTracker.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/RestrictionTracker.java
@@ -21,7 +21,7 @@ import org.apache.beam.sdk.transforms.DoFn;
/**
* Manages concurrent access to the restriction and keeps track of its claimed part for a <a
- * href="https://s.apache.org/splittable-do-fn>splittable</a> {@link DoFn}.
+ * href="https://s.apache.org/splittable-do-fn">splittable</a> {@link DoFn}.
*/
public interface RestrictionTracker<RestrictionT> {
/**
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/package-info.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/package-info.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/package-info.java
index 1ceb880..4523032 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/package-info.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/package-info.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
/**
- * Defines utilities related to <a href="https://s.apache.org/splittable-do-fn>splittable</a>
- * {@link org.apache.beam.sdk.transforms.DoFn}'s.
+ * Defines utilities related to <a href="https://s.apache.org/splittable-do-fn">splittable</a>
+ * {@link org.apache.beam.sdk.transforms.DoFn}.
*/
package org.apache.beam.sdk.transforms.splittabledofn;
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
index 45bbe75..e26f2b0 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BaseExecutionContext.java
@@ -38,7 +38,7 @@ import org.apache.beam.sdk.values.TupleTag;
* of {@link StepContext} from {@link #getOrCreateStepContext(String, String)} and
* {@link #getAllStepContexts()} without forcing each subclass to override the method, e.g.
* <pre>{@code
- * @Override
+ * {@literal @}Override
* StreamingModeExecutionContext.StepContext getOrCreateStepContext(...) {
* return (StreamingModeExecutionContext.StepContext) super.getOrCreateStepContext(...);
* }
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java
index c17d92d..b0784ca 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java
@@ -49,7 +49,7 @@ import org.apache.beam.sdk.coders.Coder.Context;
* </pre>
*
* <p>To read this stream:
- * <pre><code>
+ * <pre>{@code
* InputStream is = ...
* long count;
* do {
@@ -58,7 +58,7 @@ import org.apache.beam.sdk.coders.Coder.Context;
* // read an element from is
* }
* } while(count > 0);
- * </code></pre>
+ * }</pre>
*
* <p>The counts are encoded as variable length longs. See {@link VarInt#encode(long, OutputStream)}
* for more details. The end of the iterable is detected by reading a count of 0.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExecutionContext.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExecutionContext.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExecutionContext.java
index 82d900c..f2a79bd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExecutionContext.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExecutionContext.java
@@ -41,14 +41,14 @@ public interface ExecutionContext {
/**
* Hook for subclasses to implement that will be called whenever
- * {@link OldDoFn.Context#output}
+ * {@link org.apache.beam.sdk.transforms.OldDoFn.Context#output}
* is called.
*/
void noteOutput(WindowedValue<?> output);
/**
* Hook for subclasses to implement that will be called whenever
- * {@link OldDoFn.Context#sideOutput}
+ * {@link org.apache.beam.sdk.transforms.OldDoFn.Context#sideOutput}
* is called.
*/
void noteSideOutput(TupleTag<?> tag, WindowedValue<?> output);
@@ -70,14 +70,14 @@ public interface ExecutionContext {
/**
* Hook for subclasses to implement that will be called whenever
- * {@link OldDoFn.Context#output}
+ * {@link org.apache.beam.sdk.transforms.OldDoFn.Context#output}
* is called.
*/
void noteOutput(WindowedValue<?> output);
/**
* Hook for subclasses to implement that will be called whenever
- * {@link OldDoFn.Context#sideOutput}
+ * {@link org.apache.beam.sdk.transforms.OldDoFn.Context#sideOutput}
* is called.
*/
void noteSideOutput(TupleTag<?> tag, WindowedValue<?> output);
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayInputStream.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayInputStream.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayInputStream.java
index 020c453..662ea38 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayInputStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayInputStream.java
@@ -30,8 +30,9 @@ public class ExposedByteArrayInputStream extends ByteArrayInputStream{
super(buf);
}
- /** Read all remaining bytes.
- * @throws IOException */
+ /**
+ * Read all remaining bytes.
+ */
@SuppressFBWarnings(value = "EI_EXPOSE_REP", justification = "Returns internal buffer by design")
public byte[] readAll() throws IOException {
if (pos == 0 && count == buf.length) {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStream.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStream.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStream.java
index e2c7e42..e4a8062 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStream.java
@@ -62,8 +62,6 @@ public class ExposedByteArrayOutputStream extends ByteArrayOutputStream {
* no content copy will be involved.
*
* <p><i>Note: After passing any byte array to this method, it must not be modified again.</i>
- *
- * @throws IOException
*/
@SuppressFBWarnings(value = "EI_EXPOSE_REP", justification = "Takes ownership of input buffer")
public void writeAndOwn(byte[] b) throws IOException {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
index ce4604b..54a963b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
@@ -301,7 +301,6 @@ public class GcsUtil {
*
* @param path the GCS filename to read from
* @return a SeekableByteChannel that can read the object data
- * @throws IOException
*/
public SeekableByteChannel open(GcsPath path)
throws IOException {
@@ -319,7 +318,6 @@ public class GcsUtil {
* @param path the GCS file to write to
* @param type the type of object, eg "text/plain".
* @return a Callable object that encloses the operation.
- * @throws IOException
*/
public WritableByteChannel create(GcsPath path,
String type) throws IOException {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java
index 7b30d34..a8239ce 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java
@@ -25,8 +25,8 @@ import org.apache.beam.sdk.transforms.Combine;
/**
* Keep track of the moving minimum/maximum/sum of sampled long values. The minimum/maximum/sum
- * is over at most the last {@link #samplePeriodMs}, and is updated every
- * {@link #sampleUpdateMs}.
+ * is over at most the user-specified last {@code samplePeriodMs}, and is updated every
+ * {@code sampleUpdateMs}.
*/
public class MovingFunction {
/**
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubClient.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubClient.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubClient.java
index 1ac5511..06b776b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubClient.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubClient.java
@@ -90,10 +90,10 @@ public abstract class PubsubClient implements Closeable {
* <p>If {@code timestampLabel} is non-{@literal null} then the message attributes must contain
* that label, and the value of that label will be taken as the timestamp.
* Otherwise the timestamp will be taken from the Pubsub publish timestamp {@code
- * pubsubTimestamp}. Throw {@link IllegalArgumentException} if the timestamp cannot be
- * recognized as a ms-since-unix-epoch or RFC3339 time.
+ * pubsubTimestamp}.
*
- * @throws IllegalArgumentException
+ * @throws IllegalArgumentException if the timestamp cannot be recognized as a ms-since-unix-epoch
+ * or RFC3339 time.
*/
protected static long extractTimestamp(
@Nullable String timestampLabel,
@@ -115,7 +115,7 @@ public abstract class PubsubClient implements Closeable {
"Cannot interpret value of label %s as timestamp: %s",
timestampLabel, value);
}
- return timestampMsSinceEpoch == null ? 0 : timestampMsSinceEpoch;
+ return timestampMsSinceEpoch;
}
/**
@@ -443,8 +443,6 @@ public abstract class PubsubClient implements Closeable {
/**
* Publish {@code outgoingMessages} to Pubsub {@code topic}. Return number of messages
* published.
- *
- * @throws IOException
*/
public abstract int publish(TopicPath topic, List<OutgoingMessage> outgoingMessages)
throws IOException;
@@ -454,8 +452,6 @@ public abstract class PubsubClient implements Closeable {
* Return the received messages, or empty collection if none were available. Does not
* wait for messages to arrive if {@code returnImmediately} is {@literal true}.
* Returned messages will record their request time as {@code requestTimeMsSinceEpoch}.
- *
- * @throws IOException
*/
public abstract List<IncomingMessage> pull(
long requestTimeMsSinceEpoch,
@@ -466,8 +462,6 @@ public abstract class PubsubClient implements Closeable {
/**
* Acknowldege messages from {@code subscription} with {@code ackIds}.
- *
- * @throws IOException
*/
public abstract void acknowledge(SubscriptionPath subscription, List<String> ackIds)
throws IOException;
@@ -475,8 +469,6 @@ public abstract class PubsubClient implements Closeable {
/**
* Modify the ack deadline for messages from {@code subscription} with {@code ackIds} to
* be {@code deadlineSeconds} from now.
- *
- * @throws IOException
*/
public abstract void modifyAckDeadline(
SubscriptionPath subscription, List<String> ackIds,
@@ -484,29 +476,21 @@ public abstract class PubsubClient implements Closeable {
/**
* Create {@code topic}.
- *
- * @throws IOException
*/
public abstract void createTopic(TopicPath topic) throws IOException;
/*
* Delete {@code topic}.
- *
- * @throws IOException
*/
public abstract void deleteTopic(TopicPath topic) throws IOException;
/**
* Return a list of topics for {@code project}.
- *
- * @throws IOException
*/
public abstract List<TopicPath> listTopics(ProjectPath project) throws IOException;
/**
* Create {@code subscription} to {@code topic}.
- *
- * @throws IOException
*/
public abstract void createSubscription(
TopicPath topic, SubscriptionPath subscription, int ackDeadlineSeconds) throws IOException;
@@ -514,8 +498,6 @@ public abstract class PubsubClient implements Closeable {
/**
* Create a random subscription for {@code topic}. Return the {@link SubscriptionPath}. It
* is the responsibility of the caller to later delete the subscription.
- *
- * @throws IOException
*/
public SubscriptionPath createRandomSubscription(
ProjectPath project, TopicPath topic, int ackDeadlineSeconds) throws IOException {
@@ -529,23 +511,17 @@ public abstract class PubsubClient implements Closeable {
/**
* Delete {@code subscription}.
- *
- * @throws IOException
*/
public abstract void deleteSubscription(SubscriptionPath subscription) throws IOException;
/**
* Return a list of subscriptions for {@code topic} in {@code project}.
- *
- * @throws IOException
*/
public abstract List<SubscriptionPath> listSubscriptions(ProjectPath project, TopicPath topic)
throws IOException;
/**
* Return the ack deadline, in seconds, for {@code subscription}.
- *
- * @throws IOException
*/
public abstract int ackDeadlineSeconds(SubscriptionPath subscription) throws IOException;
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java
index 1c52c1f..3ff8448 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java
@@ -119,8 +119,8 @@ public class StringUtils {
*
* <p>Examples:
* <ul>
- * <li>{@code some.package.Word.SummaryDoFn} -> "Summary"
- * <li>{@code another.package.PairingFn} -> "Pairing"
+ * <li>{@code some.package.Word.SummaryDoFn} becomes "Summary"
+ * <li>{@code another.package.PairingFn} becomes "Pairing"
* </ul>
*
* @throws IllegalArgumentException if the class is anonymous
@@ -143,8 +143,8 @@ public class StringUtils {
*
* <p>Examples:
* <ul>
- * <li>{@code some.package.Word.Summary} -> "Word.Summary"
- * <li>{@code another.package.Pairing.Bound} -> "Pairing"
+ * <li>{@code some.package.Word.Summary} becomes "Word.Summary"
+ * <li>{@code another.package.Pairing.Bound} becomes "Pairing"
* </ul>
*/
public static String approximatePTransformName(Class<?> clazz) {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java
index 8015116..743f3f7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/TimerInternals.java
@@ -128,15 +128,17 @@ public interface TimerInternals {
* </ol>
*
* <p>In pictures:
- * <pre>
+ * <pre>{@code
* | | | | |
* | | D | C | B | A
* | | | | |
* GIWM <= GOWM <= LOWM <= LIWM <= GIWM
* (next stage)
* -------------------------------------------------> event time
- * </pre>
- * where
+ * }</pre>
+ *
+ * <p>where
+ *
* <ul>
* <li> LOWM = local output water mark.
* <li> GOWM = global output water mark.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateNamespace.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateNamespace.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateNamespace.java
index 9b27b1d..04440cb 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateNamespace.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateNamespace.java
@@ -22,17 +22,17 @@ import java.io.IOException;
/**
* A namespace used for scoping state stored with {@link StateInternals}.
*
- * <p>Instances of {@code StateNamespace} are guaranteed to have a {@link #hashCode} and
- * {@link #equals} that uniquely identify the namespace.
+ * <p>Instances of {@link StateNamespace} are guaranteed to have a {@link Object#hashCode} and
+ * {@link Object#equals} that uniquely identify the namespace.
*/
public interface StateNamespace {
/**
* Return a {@link String} representation of the key. It is guaranteed that this
- * {@code String} will uniquely identify the key.
+ * {@link String} will uniquely identify the key.
*
- * <p>This will encode the actual namespace as a {@code String}. It is
- * preferable to use the {@code StateNamespace} object when possible.
+ * <p>This will encode the actual namespace as a {@link String}. It is
+ * preferable to use the {@link StateNamespace} object when possible.
*
* <p>The string produced by the standard implementations will not contain a '+' character. This
* enables adding a '+' between the actual namespace and other information, if needed, to separate
@@ -46,11 +46,11 @@ public interface StateNamespace {
void appendTo(Appendable sb) throws IOException;
/**
- * Return an {@code Object} to use as a key in a cache.
+ * Return an {@link Object} to use as a key in a cache.
*
* <p>Different namespaces may use the same key in order to be treated as a unit in the cache.
- * The {@code Object}'s {@code hashCode} and {@code equals} methods will be used to determine
- * equality.
+ * The {@link Object}'s {@link Object#hashCode} and {@link Object#equals} methods will be used to
+ * determine equality.
*/
Object getCacheKey();
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PDone.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PDone.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PDone.java
index 83d6a92..9e8cae4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PDone.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PDone.java
@@ -24,7 +24,7 @@ import org.apache.beam.sdk.transforms.PTransform;
/**
* {@link PDone} is the output of a {@link PTransform} that has a trivial result,
- * such as a {@link Write}.
+ * such as a {@link org.apache.beam.sdk.io.Write}.
*/
public class PDone extends POutputValueBase {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
----------------------------------------------------------------------
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
index 3727f92..1d1075c 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
@@ -165,7 +165,7 @@ import org.slf4j.LoggerFactory;
* read <a href="https://cloud.google.com/datastore/docs/concepts/entities">Entities, Properties,
* and Keys</a> for more information about {@code Entity} keys.
*
- * <p><h3>Permissions</h3>
+ * <h3>Permissions</h3>
* Permission requirements depend on the {@code PipelineRunner} that is used to execute the
* Dataflow job. Please refer to the documentation of corresponding {@code PipelineRunner}s for
* more details.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/717b431c/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
----------------------------------------------------------------------
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
index 945eff6..91939d4 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
@@ -40,32 +40,29 @@ import org.joda.time.Instant;
* It follows the usage conventions laid out by other *IO classes like
* BigQueryIO or PubsubIOLet's see how you can set up a simple Pipeline, which reads from Kinesis:
*
- * <pre>{@code}
+ * <pre>{@code
* p.
* apply(KinesisIO.Read.
* from("streamName", InitialPositionInStream.LATEST).
* using("AWS_KEY", _"AWS_SECRET", STREAM_REGION).
* apply( ... ) // other transformations
- *</pre>
- * </p>
+ * }</pre>
*
- * <p>
- * As you can see you need to provide 3 things:
+ * <p>As you can see you need to provide 3 things:
* <ul>
* <li>name of the stream you're going to read</li>
- * <li>position in the stream where reading should start. There are two options:</li>
+ * <li>position in the stream where reading should start. There are two options:
* <ul>
* <li>{@link InitialPositionInStream#LATEST} - reading will begin from end of the stream</li>
* <li>{@link InitialPositionInStream#TRIM_HORIZON} - reading will begin at
* the very beginning of the stream</li>
- * </ul>
- * <li>data used to initialize {@link AmazonKinesis} client</li>
+ * </ul></li>
+ * <li>data used to initialize {@link AmazonKinesis} client:
* <ul>
* <li>credentials (aws key, aws secret)</li>
* <li>region where the stream is located</li>
- * </ul>
+ * </ul></li>
* </ul>
- * </p>
*
* <p>In case when you want to set up {@link AmazonKinesis} client by your own
* (for example if you're using more sophisticated authorization methods like Amazon STS, etc.)
@@ -73,35 +70,33 @@ import org.joda.time.Instant;
*
* <pre>{@code
* public class MyCustomKinesisClientProvider implements KinesisClientProvider {
- * @Override
+ * {@literal @}Override
* public AmazonKinesis get() {
* // set up your client here
* }
- * }}
- * </pre>
+ * }
+ * }</pre>
*
- * Usage is pretty straightforward:
+ * <p>Usage is pretty straightforward:
*
- * <pre>{@code}
+ * <pre>{@code
* p.
* apply(KinesisIO.Read.
* from("streamName", InitialPositionInStream.LATEST).
* using(MyCustomKinesisClientProvider()).
* apply( ... ) // other transformations
- * </pre>
- * </p>
+ * }</pre>
*
* <p>There\u2019s also possibility to start reading using arbitrary point in time -
* in this case you need to provide {@link Instant} object:
*
- * <pre>{@code}
+ * <pre>{@code
* p.
* apply(KinesisIO.Read.
* from("streamName", instant).
* using(MyCustomKinesisClientProvider()).
* apply( ... ) // other transformations
- * </pre>
- * </p>
+ * }</pre>
*
*/
public final class KinesisIO {
[46/50] incubator-beam git commit: Rename RegexTransform to just Regex
Posted by th...@apache.org.
Rename RegexTransform to just Regex
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6954abe4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6954abe4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6954abe4
Branch: refs/heads/apex-runner
Commit: 6954abe4e28103f5ddd3e1eebe998b765cd9de11
Parents: f6a9733
Author: Kenneth Knowles <kl...@google.com>
Authored: Mon Nov 7 10:10:32 2016 -0800
Committer: Kenneth Knowles <kl...@google.com>
Committed: Mon Nov 7 10:10:32 2016 -0800
----------------------------------------------------------------------
.../org/apache/beam/sdk/transforms/Regex.java | 505 +++++++++++++++++++
.../beam/sdk/transforms/RegexTransform.java | 505 -------------------
.../apache/beam/sdk/transforms/RegexTest.java | 262 ++++++++++
.../beam/sdk/transforms/RegexTransformTest.java | 262 ----------
4 files changed, 767 insertions(+), 767 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6954abe4/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java
new file mode 100644
index 0000000..27104f6
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java
@@ -0,0 +1,505 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+
+/**
+ * {@code PTransorm}s to use Regular Expressions to process elements in a
+ * {@link PCollection}.
+ *
+ * <p>
+ * {@link Regex#matches(String, int)} can be used to see if an entire line matches
+ * a Regex. {@link Regex#matchesKV(String, int, int)} can be used to see if an entire
+ * line matches a Regex and output certain groups as a {@link KV}.
+ * </p>
+ * <p>
+ * {@link Regex#find(String, int)} can be used to see if a portion of a line
+ * matches a Regex. {@link Regex#matchesKV(String, int, int)} can be used to see if a
+ * portion of a line matches a Regex and output certain groups as a {@link KV}.
+ * </p>
+ * <p>
+ * Lines that do not match the Regex will not be output.
+ * </p>
+ */
+public class Regex {
+ private Regex() {
+ // do not instantiate
+ }
+
+ /**
+ * Returns a {@link Regex.Matches} {@link PTransform} that checks if
+ * the entire line matches the Regex. Returns the entire line (group 0) as a
+ * {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ */
+ public static Matches matches(String regex) {
+ return matches(regex, 0);
+ }
+
+ /**
+ * Returns a {@link Regex.Matches} {@link PTransform} that checks if
+ * the entire line matches the Regex. Returns the group as a
+ * {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param group
+ * The Regex group to return as a PCollection
+ */
+ public static Matches matches(String regex, int group) {
+ return new Matches(regex, group);
+ }
+
+ /**
+ * Returns a {@link Regex.MatchesKV} {@link PTransform} that checks
+ * if the entire line matches the Regex. Returns the specified groups as the
+ * key and value as a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param keyGroup
+ * The Regex group to use as the key
+ * @param valueGroup
+ * The Regex group to use the value
+ */
+ public static MatchesKV matchesKV(String regex, int keyGroup,
+ int valueGroup) {
+ return new MatchesKV(regex, keyGroup, valueGroup);
+ }
+
+ /**
+ * Returns a {@link Regex.Find} {@link PTransform} that checks if a
+ * portion of the line matches the Regex. Returns the entire line (group 0) as
+ * a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ */
+ public static Find find(String regex) {
+ return find(regex, 0);
+ }
+
+ /**
+ * Returns a {@link Regex.Find} {@link PTransform} that checks if a
+ * portion of the line matches the Regex. Returns the group as a
+ * {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param group
+ * The Regex group to return as a PCollection
+ */
+ public static Find find(String regex, int group) {
+ return new Find(regex, group);
+ }
+
+ /**
+ * Returns a {@link Regex.FindKV} {@link PTransform} that checks if a
+ * portion of the line matches the Regex. Returns the specified groups as the
+ * key and value as a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param keyGroup
+ * The Regex group to use as the key
+ * @param valueGroup
+ * The Regex group to use the value
+ */
+ public static FindKV findKV(String regex, int keyGroup, int valueGroup) {
+ return new FindKV(regex, keyGroup, valueGroup);
+ }
+
+ /**
+ * Returns a {@link Regex.ReplaceAll} {@link PTransform} that checks if a
+ * portion of the line matches the Regex and replaces all matches with the replacement
+ * String. Returns the group as a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param replacement
+ * The string to be substituted for each match
+ */
+ public static ReplaceAll replaceAll(String regex, String replacement) {
+ return new ReplaceAll(regex, replacement);
+ }
+
+ /**
+ * Returns a {@link Regex.ReplaceAll} {@link PTransform} that checks if a
+ * portion of the line matches the Regex and replaces the first match with the replacement
+ * String. Returns the group as a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param replacement
+ * The string to be substituted for each match
+ */
+ public static ReplaceFirst replaceFirst(String regex, String replacement) {
+ return new ReplaceFirst(regex, replacement);
+ }
+
+ /**
+ * Returns a {@link Regex.Split} {@link PTransform} that splits a string
+ * on the regular expression and then outputs each item. It will not output empty
+ * items. Returns the group as a {@link PCollection}.
+ * a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ */
+ public static Split split(String regex) {
+ return split(regex, false);
+ }
+
+ /**
+ * Returns a {@link Regex.Split} {@link PTransform} that splits a string
+ * on the regular expression and then outputs each item. Returns the group as a
+ * {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param outputEmpty
+ * Should empty be output. True to output empties and false if not.
+ */
+ public static Split split(String regex, boolean outputEmpty) {
+ return new Split(regex, outputEmpty);
+ }
+
+ /**
+ * {@code Regex.Matches<String>} takes a {@code PCollection<String>}
+ * and returns a {@code PCollection<String>} representing the value
+ * extracted from the Regex groups of the input {@code PCollection}
+ * to the number of times that element occurs in the input.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If the entire line
+ * does not match the Regex, the line will not be output. If it does match the
+ * entire line, the group in the Regex will be used. The output will be the
+ * Regex group.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(Regex.matches("myregex (mygroup)", 1));
+ * }
+ * </pre>
+ */
+ public static class Matches
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ int group;
+
+ public Matches(String regex, int group) {
+ this.pattern = Pattern.compile(regex);
+ this.group = group;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in
+ .apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.matches()) {
+ c.output(m.group(group));
+ }
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code Regex.MatchesKV<KV<String, String>>} takes a
+ * {@code PCollection<String>} and returns a
+ * {@code PCollection<KV<String, String>>} representing the key and value
+ * extracted from the Regex groups of the input {@code PCollection} to the
+ * number of times that element occurs in the input.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If the entire line
+ * does not match the Regex, the line will not be output. If it does match the
+ * entire line, the groups in the Regex will be used. The key will be the
+ * key's group and the value will be the value's group.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<KV<String, String>> keysAndValues =
+ * words.apply(Regex.matchesKV("myregex (mykeygroup) (myvaluegroup)", 1, 2));
+ * }
+ * </pre>
+ */
+ public static class MatchesKV
+ extends PTransform<PCollection<String>, PCollection<KV<String, String>>> {
+ Pattern pattern;
+ int keyGroup, valueGroup;
+
+ public MatchesKV(String regex, int keyGroup, int valueGroup) {
+ this.pattern = Pattern.compile(regex);
+ this.keyGroup = keyGroup;
+ this.valueGroup = valueGroup;
+ }
+
+ public PCollection<KV<String, String>> apply(PCollection<String> in) {
+ return in.apply(ParDo
+ .of(new DoFn<String, KV<String, String>>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
+ }
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code Regex.Find<String>} takes a {@code PCollection<String>} and
+ * returns a {@code PCollection<String>} representing the value extracted
+ * from the Regex groups of the input {@code PCollection} to
+ * the number of times that element occurs in the input.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If a portion of the
+ * line does not match the Regex, the line will not be output. If it does
+ * match a portion of the line, the group in the Regex will be used. The
+ * output will be the Regex group.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(Regex.find("myregex (mygroup)", 1));
+ * }
+ * </pre>
+ */
+ public static class Find
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ int group;
+
+ public Find(String regex, int group) {
+ this.pattern = Pattern.compile(regex);
+ this.group = group;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in.apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(m.group(group));
+ }
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code Regex.MatchesKV<KV<String, String>>} takes a
+ * {@code PCollection<String>} and returns a
+ * {@code PCollection<KV<String, String>>} representing the key and value
+ * extracted from the Regex groups of the input {@code PCollection} to the
+ * number of times that element occurs in the input.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If a portion of the
+ * line does not match the Regex, the line will not be output. If it does
+ * match a portion of the line, the groups in the Regex will be used. The key
+ * will be the key's group and the value will be the value's group.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<KV<String, String>> keysAndValues =
+ * words.apply(Regex.findKV("myregex (mykeygroup) (myvaluegroup)", 1, 2));
+ * }
+ * </pre>
+ */
+ public static class FindKV
+ extends PTransform<PCollection<String>, PCollection<KV<String, String>>> {
+ Pattern pattern;
+ int keyGroup, valueGroup;
+
+ public FindKV(String regex, int keyGroup, int valueGroup) {
+ this.pattern = Pattern.compile(regex);
+ this.keyGroup = keyGroup;
+ this.valueGroup = valueGroup;
+ }
+
+ public PCollection<KV<String, String>> apply(PCollection<String> in) {
+ return in.apply(
+ ParDo.of(new DoFn<String, KV<String, String>>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
+ }
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code Regex.ReplaceAll<String>} takes a {@code PCollection<String>} and
+ * returns a {@code PCollection<String>} with all Strings that matched the
+ * Regex being replaced with the replacement string.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If a portion of the
+ * line does not match the Regex, the line will be output without changes. If it does
+ * match a portion of the line, all portions matching the Regex will be replaced
+ * with the replacement String.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(Regex.replaceAll("myregex", "myreplacement"));
+ * }
+ * </pre>
+ */
+ public static class ReplaceAll
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ String replacement;
+
+ public ReplaceAll(String regex, String replacement) {
+ this.pattern = Pattern.compile(regex);
+ this.replacement = replacement;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in.apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+ c.output(m.replaceAll(replacement));
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code Regex.ReplaceFirst<String>} takes a {@code PCollection<String>} and
+ * returns a {@code PCollection<String>} with the first Strings that matched the
+ * Regex being replaced with the replacement string.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If a portion of the
+ * line does not match the Regex, the line will be output without changes. If it does
+ * match a portion of the line, the first portion matching the Regex will be replaced
+ * with the replacement String.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(Regex.replaceFirst("myregex", "myreplacement"));
+ * }
+ * </pre>
+ */
+ public static class ReplaceFirst
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ String replacement;
+
+ public ReplaceFirst(String regex, String replacement) {
+ this.pattern = Pattern.compile(regex);
+ this.replacement = replacement;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in.apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+ c.output(m.replaceFirst(replacement));
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code Regex.Split<String>} takes a {@code PCollection<String>} and
+ * returns a {@code PCollection<String>} with the input string split into
+ * individual items in a list. Each item is then output as a separate string.
+ *
+ * <p>
+ * This transform runs a Regex as part of a splint the entire input line. The split
+ * gives back an array of items. Each item is output as a separate item in the
+ * {@code PCollection<String>}.
+ * </p>
+ *
+ * <p>
+ * Depending on the Regex, a split can be an empty or
+ * "" string. You can pass in a parameter if you want empty strings or not.
+ * </p>
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(Regex.split("\W*"));
+ * }
+ * </pre>
+ */
+ public static class Split
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ boolean outputEmpty;
+
+ public Split(String regex, boolean outputEmpty) {
+ this.pattern = Pattern.compile(regex);
+ this.outputEmpty = outputEmpty;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in.apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ String[] items = pattern.split(c.element());
+
+ for (String item : items) {
+ if (outputEmpty || !item.isEmpty()) {
+ c.output(item);
+ }
+ }
+ }
+ }));
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6954abe4/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/RegexTransform.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/RegexTransform.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/RegexTransform.java
deleted file mode 100644
index bd7848a..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/RegexTransform.java
+++ /dev/null
@@ -1,505 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms;
-
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-
-/**
- * {@code PTransorm}s to use Regular Expressions to process elements in a
- * {@link PCollection}.
- *
- * <p>
- * {@link RegexTransform#matches(String, int)} can be used to see if an entire line matches
- * a Regex. {@link RegexTransform#matchesKV(String, int, int)} can be used to see if an entire
- * line matches a Regex and output certain groups as a {@link KV}.
- * </p>
- * <p>
- * {@link RegexTransform#find(String, int)} can be used to see if a portion of a line
- * matches a Regex. {@link RegexTransform#matchesKV(String, int, int)} can be used to see if a
- * portion of a line matches a Regex and output certain groups as a {@link KV}.
- * </p>
- * <p>
- * Lines that do not match the Regex will not be output.
- * </p>
- */
-public class RegexTransform {
- private RegexTransform() {
- // do not instantiate
- }
-
- /**
- * Returns a {@link RegexTransform.Matches} {@link PTransform} that checks if
- * the entire line matches the Regex. Returns the entire line (group 0) as a
- * {@link PCollection}.
- * @param regex
- * The regular expression to run
- */
- public static Matches matches(String regex) {
- return matches(regex, 0);
- }
-
- /**
- * Returns a {@link RegexTransform.Matches} {@link PTransform} that checks if
- * the entire line matches the Regex. Returns the group as a
- * {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param group
- * The Regex group to return as a PCollection
- */
- public static Matches matches(String regex, int group) {
- return new Matches(regex, group);
- }
-
- /**
- * Returns a {@link RegexTransform.MatchesKV} {@link PTransform} that checks
- * if the entire line matches the Regex. Returns the specified groups as the
- * key and value as a {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param keyGroup
- * The Regex group to use as the key
- * @param valueGroup
- * The Regex group to use the value
- */
- public static MatchesKV matchesKV(String regex, int keyGroup,
- int valueGroup) {
- return new MatchesKV(regex, keyGroup, valueGroup);
- }
-
- /**
- * Returns a {@link RegexTransform.Find} {@link PTransform} that checks if a
- * portion of the line matches the Regex. Returns the entire line (group 0) as
- * a {@link PCollection}.
- * @param regex
- * The regular expression to run
- */
- public static Find find(String regex) {
- return find(regex, 0);
- }
-
- /**
- * Returns a {@link RegexTransform.Find} {@link PTransform} that checks if a
- * portion of the line matches the Regex. Returns the group as a
- * {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param group
- * The Regex group to return as a PCollection
- */
- public static Find find(String regex, int group) {
- return new Find(regex, group);
- }
-
- /**
- * Returns a {@link RegexTransform.FindKV} {@link PTransform} that checks if a
- * portion of the line matches the Regex. Returns the specified groups as the
- * key and value as a {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param keyGroup
- * The Regex group to use as the key
- * @param valueGroup
- * The Regex group to use the value
- */
- public static FindKV findKV(String regex, int keyGroup, int valueGroup) {
- return new FindKV(regex, keyGroup, valueGroup);
- }
-
- /**
- * Returns a {@link RegexTransform.ReplaceAll} {@link PTransform} that checks if a
- * portion of the line matches the Regex and replaces all matches with the replacement
- * String. Returns the group as a {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param replacement
- * The string to be substituted for each match
- */
- public static ReplaceAll replaceAll(String regex, String replacement) {
- return new ReplaceAll(regex, replacement);
- }
-
- /**
- * Returns a {@link RegexTransform.ReplaceAll} {@link PTransform} that checks if a
- * portion of the line matches the Regex and replaces the first match with the replacement
- * String. Returns the group as a {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param replacement
- * The string to be substituted for each match
- */
- public static ReplaceFirst replaceFirst(String regex, String replacement) {
- return new ReplaceFirst(regex, replacement);
- }
-
- /**
- * Returns a {@link RegexTransform.Split} {@link PTransform} that splits a string
- * on the regular expression and then outputs each item. It will not output empty
- * items. Returns the group as a {@link PCollection}.
- * a {@link PCollection}.
- * @param regex
- * The regular expression to run
- */
- public static Split split(String regex) {
- return split(regex, false);
- }
-
- /**
- * Returns a {@link RegexTransform.Split} {@link PTransform} that splits a string
- * on the regular expression and then outputs each item. Returns the group as a
- * {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param outputEmpty
- * Should empty be output. True to output empties and false if not.
- */
- public static Split split(String regex, boolean outputEmpty) {
- return new Split(regex, outputEmpty);
- }
-
- /**
- * {@code RegexTransform.Matches<String>} takes a {@code PCollection<String>}
- * and returns a {@code PCollection<String>} representing the value
- * extracted from the Regex groups of the input {@code PCollection}
- * to the number of times that element occurs in the input.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If the entire line
- * does not match the Regex, the line will not be output. If it does match the
- * entire line, the group in the Regex will be used. The output will be the
- * Regex group.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
- * PCollection<String> words = ...;
- * PCollection<String> values =
- * words.apply(RegexTransform.matches("myregex (mygroup)", 1));
- * }
- * </pre>
- */
- public static class Matches
- extends PTransform<PCollection<String>, PCollection<String>> {
- Pattern pattern;
- int group;
-
- public Matches(String regex, int group) {
- this.pattern = Pattern.compile(regex);
- this.group = group;
- }
-
- public PCollection<String> apply(PCollection<String> in) {
- return in
- .apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
-
- if (m.matches()) {
- c.output(m.group(group));
- }
- }
- }));
- }
- }
-
- /**
- * {@code RegexTransform.MatchesKV<KV<String, String>>} takes a
- * {@code PCollection<String>} and returns a
- * {@code PCollection<KV<String, String>>} representing the key and value
- * extracted from the Regex groups of the input {@code PCollection} to the
- * number of times that element occurs in the input.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If the entire line
- * does not match the Regex, the line will not be output. If it does match the
- * entire line, the groups in the Regex will be used. The key will be the
- * key's group and the value will be the value's group.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
- * PCollection<String> words = ...;
- * PCollection<KV<String, String>> keysAndValues =
- * words.apply(RegexTransform.matchesKV("myregex (mykeygroup) (myvaluegroup)", 1, 2));
- * }
- * </pre>
- */
- public static class MatchesKV
- extends PTransform<PCollection<String>, PCollection<KV<String, String>>> {
- Pattern pattern;
- int keyGroup, valueGroup;
-
- public MatchesKV(String regex, int keyGroup, int valueGroup) {
- this.pattern = Pattern.compile(regex);
- this.keyGroup = keyGroup;
- this.valueGroup = valueGroup;
- }
-
- public PCollection<KV<String, String>> apply(PCollection<String> in) {
- return in.apply(ParDo
- .of(new DoFn<String, KV<String, String>>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
-
- if (m.find()) {
- c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
- }
- }
- }));
- }
- }
-
- /**
- * {@code RegexTransform.Find<String>} takes a {@code PCollection<String>} and
- * returns a {@code PCollection<String>} representing the value extracted
- * from the Regex groups of the input {@code PCollection} to
- * the number of times that element occurs in the input.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If a portion of the
- * line does not match the Regex, the line will not be output. If it does
- * match a portion of the line, the group in the Regex will be used. The
- * output will be the Regex group.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
- * PCollection<String> words = ...;
- * PCollection<String> values =
- * words.apply(RegexTransform.find("myregex (mygroup)", 1));
- * }
- * </pre>
- */
- public static class Find
- extends PTransform<PCollection<String>, PCollection<String>> {
- Pattern pattern;
- int group;
-
- public Find(String regex, int group) {
- this.pattern = Pattern.compile(regex);
- this.group = group;
- }
-
- public PCollection<String> apply(PCollection<String> in) {
- return in.apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
-
- if (m.find()) {
- c.output(m.group(group));
- }
- }
- }));
- }
- }
-
- /**
- * {@code RegexTransform.MatchesKV<KV<String, String>>} takes a
- * {@code PCollection<String>} and returns a
- * {@code PCollection<KV<String, String>>} representing the key and value
- * extracted from the Regex groups of the input {@code PCollection} to the
- * number of times that element occurs in the input.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If a portion of the
- * line does not match the Regex, the line will not be output. If it does
- * match a portion of the line, the groups in the Regex will be used. The key
- * will be the key's group and the value will be the value's group.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
- * PCollection<String> words = ...;
- * PCollection<KV<String, String>> keysAndValues =
- * words.apply(RegexTransform.findKV("myregex (mykeygroup) (myvaluegroup)", 1, 2));
- * }
- * </pre>
- */
- public static class FindKV
- extends PTransform<PCollection<String>, PCollection<KV<String, String>>> {
- Pattern pattern;
- int keyGroup, valueGroup;
-
- public FindKV(String regex, int keyGroup, int valueGroup) {
- this.pattern = Pattern.compile(regex);
- this.keyGroup = keyGroup;
- this.valueGroup = valueGroup;
- }
-
- public PCollection<KV<String, String>> apply(PCollection<String> in) {
- return in.apply(
- ParDo.of(new DoFn<String, KV<String, String>>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
-
- if (m.find()) {
- c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
- }
- }
- }));
- }
- }
-
- /**
- * {@code RegexTransform.ReplaceAll<String>} takes a {@code PCollection<String>} and
- * returns a {@code PCollection<String>} with all Strings that matched the
- * Regex being replaced with the replacement string.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If a portion of the
- * line does not match the Regex, the line will be output without changes. If it does
- * match a portion of the line, all portions matching the Regex will be replaced
- * with the replacement String.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
- * PCollection<String> words = ...;
- * PCollection<String> values =
- * words.apply(RegexTransform.replaceAll("myregex", "myreplacement"));
- * }
- * </pre>
- */
- public static class ReplaceAll
- extends PTransform<PCollection<String>, PCollection<String>> {
- Pattern pattern;
- String replacement;
-
- public ReplaceAll(String regex, String replacement) {
- this.pattern = Pattern.compile(regex);
- this.replacement = replacement;
- }
-
- public PCollection<String> apply(PCollection<String> in) {
- return in.apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
- c.output(m.replaceAll(replacement));
- }
- }));
- }
- }
-
- /**
- * {@code RegexTransform.ReplaceFirst<String>} takes a {@code PCollection<String>} and
- * returns a {@code PCollection<String>} with the first Strings that matched the
- * Regex being replaced with the replacement string.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If a portion of the
- * line does not match the Regex, the line will be output without changes. If it does
- * match a portion of the line, the first portion matching the Regex will be replaced
- * with the replacement String.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
- * PCollection<String> words = ...;
- * PCollection<String> values =
- * words.apply(RegexTransform.replaceFirst("myregex", "myreplacement"));
- * }
- * </pre>
- */
- public static class ReplaceFirst
- extends PTransform<PCollection<String>, PCollection<String>> {
- Pattern pattern;
- String replacement;
-
- public ReplaceFirst(String regex, String replacement) {
- this.pattern = Pattern.compile(regex);
- this.replacement = replacement;
- }
-
- public PCollection<String> apply(PCollection<String> in) {
- return in.apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
- c.output(m.replaceFirst(replacement));
- }
- }));
- }
- }
-
- /**
- * {@code RegexTransform.Split<String>} takes a {@code PCollection<String>} and
- * returns a {@code PCollection<String>} with the input string split into
- * individual items in a list. Each item is then output as a separate string.
- *
- * <p>
- * This transform runs a Regex as part of a splint the entire input line. The split
- * gives back an array of items. Each item is output as a separate item in the
- * {@code PCollection<String>}.
- * </p>
- *
- * <p>
- * Depending on the Regex, a split can be an empty or
- * "" string. You can pass in a parameter if you want empty strings or not.
- * </p>
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
- * PCollection<String> words = ...;
- * PCollection<String> values =
- * words.apply(RegexTransform.split("\W*"));
- * }
- * </pre>
- */
- public static class Split
- extends PTransform<PCollection<String>, PCollection<String>> {
- Pattern pattern;
- boolean outputEmpty;
-
- public Split(String regex, boolean outputEmpty) {
- this.pattern = Pattern.compile(regex);
- this.outputEmpty = outputEmpty;
- }
-
- public PCollection<String> apply(PCollection<String> in) {
- return in.apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- String[] items = pattern.split(c.element());
-
- for (String item : items) {
- if (outputEmpty || !item.isEmpty()) {
- c.output(item);
- }
- }
- }
- }));
- }
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6954abe4/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java
new file mode 100644
index 0000000..71f080e
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import java.io.Serializable;
+
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link Regex}.
+ */
+@RunWith(JUnit4.class)
+public class RegexTest implements Serializable {
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFind() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("aj", "xj", "yj", "zj"))
+ .apply(Regex.find("[xyz]"));
+
+ PAssert.that(output).containsInAnyOrder("x", "y", "z");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFindGroup() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("aj", "xj", "yj", "zj"))
+ .apply(Regex.find("([xyz])", 1));
+
+ PAssert.that(output).containsInAnyOrder("x", "y", "z");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFindNone() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("a", "b", "c", "d"))
+ .apply(Regex.find("[xyz]"));
+
+ PAssert.that(output).empty();
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testKVFind() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<KV<String, String>> output = p
+ .apply(Create.of("a b c"))
+ .apply(Regex.findKV("a (b) (c)", 1, 2));
+
+ PAssert.that(output).containsInAnyOrder(KV.of("b", "c"));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testKVFindNone() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<KV<String, String>> output = p
+ .apply(Create.of("x y z"))
+ .apply(Regex.findKV("a (b) (c)", 1, 2));
+
+ PAssert.that(output).empty();
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMatches() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("a", "x", "y", "z"))
+ .apply(Regex.matches("[xyz]"));
+
+ PAssert.that(output).containsInAnyOrder("x", "y", "z");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMatchesNone() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("a", "b", "c", "d"))
+ .apply(Regex.matches("[xyz]"));
+
+ PAssert.that(output).empty();
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMatchesGroup() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("a", "x xxx", "x yyy", "x zzz"))
+ .apply(Regex.matches("x ([xyz]*)", 1));
+
+ PAssert.that(output).containsInAnyOrder("xxx", "yyy", "zzz");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testKVMatches() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<KV<String, String>> output = p
+ .apply(Create.of("a b c"))
+ .apply(Regex.matchesKV("a (b) (c)", 1, 2));
+
+ PAssert.that(output).containsInAnyOrder(KV.of("b", "c"));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testKVMatchesNone() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<KV<String, String>> output = p
+ .apply(Create.of("x y z"))
+ .apply(Regex.matchesKV("a (b) (c)", 1, 2));
+ PAssert.that(output).empty();
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testReplaceAll() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("xj", "yj", "zj"))
+ .apply(Regex.replaceAll("[xyz]", "new"));
+
+ PAssert.that(output).containsInAnyOrder("newj", "newj", "newj");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testReplaceAllMixed() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("abc", "xj", "yj", "zj", "def"))
+ .apply(Regex.replaceAll("[xyz]", "new"));
+
+ PAssert.that(output).containsInAnyOrder("abc", "newj", "newj", "newj", "def");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testReplaceFirst() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("xjx", "yjy", "zjz"))
+ .apply(Regex.replaceFirst("[xyz]", "new"));
+
+ PAssert.that(output).containsInAnyOrder("newjx", "newjy", "newjz");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testReplaceFirstMixed() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("abc", "xjx", "yjy", "zjz", "def"))
+ .apply(Regex.replaceFirst("[xyz]", "new"));
+
+ PAssert.that(output).containsInAnyOrder("abc", "newjx", "newjy", "newjz", "def");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSplits() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(Regex.split("\\W+"));
+
+ PAssert.that(output).containsInAnyOrder("The", "quick", "brown",
+ "fox", "jumps", "over", "the", "lazy", "dog");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSplitsWithEmpty() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(Regex.split("\\s", true));
+
+ String[] outputStr = "The quick brown fox jumps over the lazy dog".split("\\s");
+
+ PAssert.that(output).containsInAnyOrder("The", "", "quick", "brown", "", "",
+ "fox", "jumps", "over", "", "", "", "the", "lazy", "dog");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSplitsWithoutEmpty() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(Regex.split("\\s", false));
+
+ PAssert.that(output).containsInAnyOrder("The", "quick", "brown",
+ "fox", "jumps", "over", "the", "lazy", "dog");
+ p.run();
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6954abe4/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTransformTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTransformTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTransformTest.java
deleted file mode 100644
index 63d02d7..0000000
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTransformTest.java
+++ /dev/null
@@ -1,262 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms;
-
-import java.io.Serializable;
-
-import org.apache.beam.sdk.testing.NeedsRunner;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-
-import org.junit.Test;
-import org.junit.experimental.categories.Category;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/**
- * Tests for {@link RegexTransform}.
- */
-@RunWith(JUnit4.class)
-public class RegexTransformTest implements Serializable {
- @Test
- @Category(NeedsRunner.class)
- public void testFind() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("aj", "xj", "yj", "zj"))
- .apply(RegexTransform.find("[xyz]"));
-
- PAssert.that(output).containsInAnyOrder("x", "y", "z");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testFindGroup() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("aj", "xj", "yj", "zj"))
- .apply(RegexTransform.find("([xyz])", 1));
-
- PAssert.that(output).containsInAnyOrder("x", "y", "z");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testFindNone() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("a", "b", "c", "d"))
- .apply(RegexTransform.find("[xyz]"));
-
- PAssert.that(output).empty();
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testKVFind() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<KV<String, String>> output = p
- .apply(Create.of("a b c"))
- .apply(RegexTransform.findKV("a (b) (c)", 1, 2));
-
- PAssert.that(output).containsInAnyOrder(KV.of("b", "c"));
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testKVFindNone() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<KV<String, String>> output = p
- .apply(Create.of("x y z"))
- .apply(RegexTransform.findKV("a (b) (c)", 1, 2));
-
- PAssert.that(output).empty();
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testMatches() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("a", "x", "y", "z"))
- .apply(RegexTransform.matches("[xyz]"));
-
- PAssert.that(output).containsInAnyOrder("x", "y", "z");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testMatchesNone() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("a", "b", "c", "d"))
- .apply(RegexTransform.matches("[xyz]"));
-
- PAssert.that(output).empty();
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testMatchesGroup() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("a", "x xxx", "x yyy", "x zzz"))
- .apply(RegexTransform.matches("x ([xyz]*)", 1));
-
- PAssert.that(output).containsInAnyOrder("xxx", "yyy", "zzz");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testKVMatches() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<KV<String, String>> output = p
- .apply(Create.of("a b c"))
- .apply(RegexTransform.matchesKV("a (b) (c)", 1, 2));
-
- PAssert.that(output).containsInAnyOrder(KV.of("b", "c"));
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testKVMatchesNone() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<KV<String, String>> output = p
- .apply(Create.of("x y z"))
- .apply(RegexTransform.matchesKV("a (b) (c)", 1, 2));
- PAssert.that(output).empty();
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testReplaceAll() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("xj", "yj", "zj"))
- .apply(RegexTransform.replaceAll("[xyz]", "new"));
-
- PAssert.that(output).containsInAnyOrder("newj", "newj", "newj");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testReplaceAllMixed() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("abc", "xj", "yj", "zj", "def"))
- .apply(RegexTransform.replaceAll("[xyz]", "new"));
-
- PAssert.that(output).containsInAnyOrder("abc", "newj", "newj", "newj", "def");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testReplaceFirst() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("xjx", "yjy", "zjz"))
- .apply(RegexTransform.replaceFirst("[xyz]", "new"));
-
- PAssert.that(output).containsInAnyOrder("newjx", "newjy", "newjz");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testReplaceFirstMixed() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("abc", "xjx", "yjy", "zjz", "def"))
- .apply(RegexTransform.replaceFirst("[xyz]", "new"));
-
- PAssert.that(output).containsInAnyOrder("abc", "newjx", "newjy", "newjz", "def");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testSplits() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("The quick brown fox jumps over the lazy dog"))
- .apply(RegexTransform.split("\\W+"));
-
- PAssert.that(output).containsInAnyOrder("The", "quick", "brown",
- "fox", "jumps", "over", "the", "lazy", "dog");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testSplitsWithEmpty() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("The quick brown fox jumps over the lazy dog"))
- .apply(RegexTransform.split("\\s", true));
-
- String[] outputStr = "The quick brown fox jumps over the lazy dog".split("\\s");
-
- PAssert.that(output).containsInAnyOrder("The", "", "quick", "brown", "", "",
- "fox", "jumps", "over", "", "", "", "the", "lazy", "dog");
- p.run();
- }
-
- @Test
- @Category(NeedsRunner.class)
- public void testSplitsWithoutEmpty() {
- TestPipeline p = TestPipeline.create();
-
- PCollection<String> output = p
- .apply(Create.of("The quick brown fox jumps over the lazy dog"))
- .apply(RegexTransform.split("\\s", false));
-
- PAssert.that(output).containsInAnyOrder("The", "quick", "brown",
- "fox", "jumps", "over", "the", "lazy", "dog");
- p.run();
- }
-}
[48/50] incubator-beam git commit: This closes #1296
Posted by th...@apache.org.
This closes #1296
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/11024552
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/11024552
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/11024552
Branch: refs/heads/apex-runner
Commit: 110245526d204287ffb30d9b8e91aca4003542f6
Parents: f6a9733 79b0455
Author: Kenneth Knowles <kl...@google.com>
Authored: Mon Nov 7 10:28:21 2016 -0800
Committer: Kenneth Knowles <kl...@google.com>
Committed: Mon Nov 7 10:28:21 2016 -0800
----------------------------------------------------------------------
.../org/apache/beam/sdk/transforms/Regex.java | 454 +++++++++++++++++
.../beam/sdk/transforms/RegexTransform.java | 505 -------------------
.../apache/beam/sdk/transforms/RegexTest.java | 248 +++++++++
.../beam/sdk/transforms/RegexTransformTest.java | 262 ----------
4 files changed, 702 insertions(+), 767 deletions(-)
----------------------------------------------------------------------
[36/50] incubator-beam git commit: [BEAM-898] Fix Jenkins
BigQueryTornadoes IT Failure
Posted by th...@apache.org.
[BEAM-898] Fix Jenkins BigQueryTornadoes IT Failure
This closes #1279
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/46fbfe06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/46fbfe06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/46fbfe06
Branch: refs/heads/apex-runner
Commit: 46fbfe06bc34f9c355d00f346f63767a861cb858
Parents: 99062d1 1eccd29
Author: Luke Cwik <lc...@google.com>
Authored: Fri Nov 4 14:20:14 2016 -0700
Committer: Luke Cwik <lc...@google.com>
Committed: Fri Nov 4 14:20:14 2016 -0700
----------------------------------------------------------------------
.../beam/examples/WindowedWordCountIT.java | 11 ++++++++---
.../org/apache/beam/examples/WordCountIT.java | 19 +++++++++++--------
.../examples/cookbook/BigQueryTornadoesIT.java | 11 ++++++++---
3 files changed, 27 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
[24/50] incubator-beam git commit: Refactor and reuse parameter
analysis in DoFnSignatures
Posted by th...@apache.org.
Refactor and reuse parameter analysis in DoFnSignatures
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8bf6d92c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8bf6d92c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8bf6d92c
Branch: refs/heads/apex-runner
Commit: 8bf6d92cf35d11f4f3b02dae677a4fe778d34a61
Parents: 71fa7cd
Author: Kenneth Knowles <kl...@google.com>
Authored: Mon Oct 31 21:30:40 2016 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Thu Nov 3 21:32:53 2016 -0700
----------------------------------------------------------------------
.../sdk/transforms/reflect/DoFnSignature.java | 21 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 585 ++++++++++++-------
.../DoFnSignaturesProcessElementTest.java | 18 +-
.../DoFnSignaturesSplittableDoFnTest.java | 1 -
.../transforms/reflect/DoFnSignaturesTest.java | 35 +-
.../reflect/DoFnSignaturesTestUtils.java | 5 +-
6 files changed, 446 insertions(+), 219 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8bf6d92c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index 431de02..7087efa 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -126,12 +126,25 @@ public abstract class DoFnSignature {
abstract DoFnSignature build();
}
- /** A method delegated to a annotated method of an underlying {@link DoFn}. */
+ /** A method delegated to an annotated method of an underlying {@link DoFn}. */
public interface DoFnMethod {
/** The annotated method itself. */
Method targetMethod();
}
+ /**
+ * A method delegated to an annotated method of an underlying {@link DoFn} that accepts a dynamic
+ * list of parameters.
+ */
+ public interface MethodWithExtraParameters extends DoFnMethod {
+ /**
+ * Types of optional parameters of the annotated method, in the order they appear.
+ *
+ * <p>Validation that these are allowed is external to this class.
+ */
+ List<Parameter> extraParameters();
+ }
+
/** A descriptor for an optional parameter of the {@link DoFn.ProcessElement} method. */
public abstract static class Parameter {
@@ -331,12 +344,13 @@ public abstract class DoFnSignature {
/** Describes a {@link DoFn.ProcessElement} method. */
@AutoValue
- public abstract static class ProcessElementMethod implements DoFnMethod {
+ public abstract static class ProcessElementMethod implements MethodWithExtraParameters {
/** The annotated method itself. */
@Override
public abstract Method targetMethod();
/** Types of optional parameters of the annotated method, in the order they appear. */
+ @Override
public abstract List<Parameter> extraParameters();
/** Concrete type of the {@link RestrictionTracker} parameter, if present. */
@@ -380,7 +394,7 @@ public abstract class DoFnSignature {
/** Describes a {@link DoFn.OnTimer} method. */
@AutoValue
- public abstract static class OnTimerMethod implements DoFnMethod {
+ public abstract static class OnTimerMethod implements MethodWithExtraParameters {
/** The id on the method's {@link DoFn.TimerId} annotation. */
public abstract String id();
@@ -390,6 +404,7 @@ public abstract class DoFnSignature {
public abstract Method targetMethod();
/** Types of optional parameters of the annotated method, in the order they appear. */
+ @Override
public abstract List<Parameter> extraParameters();
static OnTimerMethod create(Method targetMethod, String id, List<Parameter> extraParameters) {
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8bf6d92c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
index c690ace..0475404 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
@@ -19,6 +19,7 @@ package org.apache.beam.sdk.transforms.reflect;
import static com.google.common.base.Preconditions.checkState;
+import com.google.auto.value.AutoValue;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Maps;
@@ -41,9 +42,11 @@ import java.util.Map;
import javax.annotation.Nullable;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.StateId;
import org.apache.beam.sdk.transforms.DoFn.TimerId;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignature.OnTimerMethod;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.StateDeclaration;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.TimerDeclaration;
import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
@@ -81,13 +84,134 @@ public class DoFnSignatures {
return signature;
}
+ /**
+ * The context for a {@link DoFn} class, for use in analysis.
+ *
+ * <p>It contains much of the information that eventually becomes part of the {@link
+ * DoFnSignature}, but in an intermediate state.
+ */
+ @VisibleForTesting
+ static class FnAnalysisContext {
+
+ private final Map<String, StateDeclaration> stateDeclarations = new HashMap<>();
+ private final Map<String, TimerDeclaration> timerDeclarations = new HashMap<>();
+
+ private FnAnalysisContext() {}
+
+ /** Create an empty context, with no declarations. */
+ public static FnAnalysisContext create() {
+ return new FnAnalysisContext();
+ }
+
+ /** State parameters declared in this context, keyed by {@link StateId}. Unmodifiable. */
+ public Map<String, StateDeclaration> getStateDeclarations() {
+ return Collections.unmodifiableMap(stateDeclarations);
+ }
+
+ /** Timer parameters declared in this context, keyed by {@link TimerId}. Unmodifiable. */
+ public Map<String, TimerDeclaration> getTimerDeclarations() {
+ return Collections.unmodifiableMap(timerDeclarations);
+ }
+
+ public void addStateDeclaration(StateDeclaration decl) {
+ stateDeclarations.put(decl.id(), decl);
+ }
+
+ public void addStateDeclarations(Iterable<StateDeclaration> decls) {
+ for (StateDeclaration decl : decls) {
+ addStateDeclaration(decl);
+ }
+ }
+
+ public void addTimerDeclaration(TimerDeclaration decl) {
+ timerDeclarations.put(decl.id(), decl);
+ }
+
+ public void addTimerDeclarations(Iterable<TimerDeclaration> decls) {
+ for (TimerDeclaration decl : decls) {
+ addTimerDeclaration(decl);
+ }
+ }
+ }
+
+ /**
+ * The context of analysis within a particular method.
+ *
+ * <p>It contains much of the information that eventually becomes part of the {@link
+ * DoFnSignature.MethodWithExtraParameters}, but in an intermediate state.
+ */
+ private static class MethodAnalysisContext {
+
+ private final Map<String, StateParameter> stateParameters = new HashMap<>();
+ private final Map<String, TimerParameter> timerParameters = new HashMap<>();
+ private final List<Parameter> extraParameters = new ArrayList<>();
+
+ private MethodAnalysisContext() {}
+
+ /** State parameters declared in this context, keyed by {@link StateId}. */
+ public Map<String, StateParameter> getStateParameters() {
+ return Collections.unmodifiableMap(stateParameters);
+ }
+
+ /** Timer parameters declared in this context, keyed by {@link TimerId}. */
+ public Map<String, TimerParameter> getTimerParameters() {
+ return Collections.unmodifiableMap(timerParameters);
+ }
+
+ /** Extra parameters in their entirety. Unmodifiable. */
+ public List<Parameter> getExtraParameters() {
+ return Collections.unmodifiableList(extraParameters);
+ }
+
+ /**
+ * Returns an {@link MethodAnalysisContext} like this one but including the provided {@link
+ * StateParameter}.
+ */
+ public void addParameter(Parameter param) {
+ extraParameters.add(param);
+
+ if (param instanceof StateParameter) {
+ StateParameter stateParameter = (StateParameter) param;
+ stateParameters.put(stateParameter.referent().id(), stateParameter);
+ }
+ if (param instanceof TimerParameter) {
+ TimerParameter timerParameter = (TimerParameter) param;
+ timerParameters.put(timerParameter.referent().id(), timerParameter);
+ }
+ }
+
+ /** Create an empty context, with no declarations. */
+ public static MethodAnalysisContext create() {
+ return new MethodAnalysisContext();
+ }
+ }
+
+ @AutoValue
+ abstract static class ParameterDescription {
+ public abstract Method getMethod();
+ public abstract int getIndex();
+ public abstract TypeDescriptor<?> getType();
+ public abstract List<Annotation> getAnnotations();
+
+ public static ParameterDescription of(
+ Method method, int index, TypeDescriptor<?> type, List<Annotation> annotations) {
+ return new AutoValue_DoFnSignatures_ParameterDescription(method, index, type, annotations);
+ }
+
+ public static ParameterDescription of(
+ Method method, int index, TypeDescriptor<?> type, Annotation[] annotations) {
+ return new AutoValue_DoFnSignatures_ParameterDescription(
+ method, index, type, Arrays.asList(annotations));
+ }
+ }
+
/** Analyzes a given {@link DoFn} class and extracts its {@link DoFnSignature}. */
private static DoFnSignature parseSignature(Class<? extends DoFn<?, ?>> fnClass) {
- DoFnSignature.Builder builder = DoFnSignature.builder();
+ DoFnSignature.Builder signatureBuilder = DoFnSignature.builder();
ErrorReporter errors = new ErrorReporter(null, fnClass.getName());
errors.checkArgument(DoFn.class.isAssignableFrom(fnClass), "Must be subtype of DoFn");
- builder.setFnClass(fnClass);
+ signatureBuilder.setFnClass(fnClass);
TypeDescriptor<? extends DoFn<?, ?>> fnT = TypeDescriptor.of(fnClass);
@@ -106,11 +230,9 @@ public class DoFnSignatures {
// Find the state and timer declarations in advance of validating
// method parameter lists
- Map<String, StateDeclaration> stateDeclarations = analyzeStateDeclarations(errors, fnClass);
- builder.setStateDeclarations(stateDeclarations);
-
- Map<String, TimerDeclaration> timerDeclarations = analyzeTimerDeclarations(errors, fnClass);
- builder.setTimerDeclarations(timerDeclarations);
+ FnAnalysisContext fnContext = FnAnalysisContext.create();
+ fnContext.addStateDeclarations(analyzeStateDeclarations(errors, fnClass).values());
+ fnContext.addTimerDeclarations(analyzeTimerDeclarations(errors, fnClass).values());
Method processElementMethod =
findAnnotatedMethod(errors, DoFn.ProcessElement.class, fnClass, true);
@@ -135,12 +257,12 @@ public class DoFnSignatures {
for (Method onTimerMethod : onTimerMethods) {
String id = onTimerMethod.getAnnotation(DoFn.OnTimer.class).value();
errors.checkArgument(
- timerDeclarations.containsKey(id),
+ fnContext.getTimerDeclarations().containsKey(id),
"Callback %s is for for undeclared timer %s",
onTimerMethod,
id);
- TimerDeclaration timerDecl = timerDeclarations.get(id);
+ TimerDeclaration timerDecl = fnContext.getTimerDeclarations().get(id);
errors.checkArgument(
timerDecl.field().getDeclaringClass().equals(onTimerMethod.getDeclaringClass()),
"Callback %s is for timer %s declared in a different class %s."
@@ -149,13 +271,14 @@ public class DoFnSignatures {
id,
timerDecl.field().getDeclaringClass().getCanonicalName());
- onTimerMethodMap.put(id, OnTimerMethod.create(onTimerMethod, id, Collections.EMPTY_LIST));
+ onTimerMethodMap.put(
+ id, analyzeOnTimerMethod(errors, fnT, onTimerMethod, id, outputT, fnContext));
}
- builder.setOnTimerMethods(onTimerMethodMap);
+ signatureBuilder.setOnTimerMethods(onTimerMethodMap);
// Check the converse - that all timers have a callback. This could be relaxed to only
// those timers used in methods, once method parameter lists support timers.
- for (TimerDeclaration decl : timerDeclarations.values()) {
+ for (TimerDeclaration decl : fnContext.getTimerDeclarations().values()) {
errors.checkArgument(
onTimerMethodMap.containsKey(decl.id()),
"No callback registered via %s for timer %s",
@@ -172,30 +295,29 @@ public class DoFnSignatures {
processElementMethod,
inputT,
outputT,
- stateDeclarations,
- timerDeclarations);
- builder.setProcessElement(processElement);
+ fnContext);
+ signatureBuilder.setProcessElement(processElement);
if (startBundleMethod != null) {
ErrorReporter startBundleErrors = errors.forMethod(DoFn.StartBundle.class, startBundleMethod);
- builder.setStartBundle(
+ signatureBuilder.setStartBundle(
analyzeBundleMethod(startBundleErrors, fnT, startBundleMethod, inputT, outputT));
}
if (finishBundleMethod != null) {
ErrorReporter finishBundleErrors =
errors.forMethod(DoFn.FinishBundle.class, finishBundleMethod);
- builder.setFinishBundle(
+ signatureBuilder.setFinishBundle(
analyzeBundleMethod(finishBundleErrors, fnT, finishBundleMethod, inputT, outputT));
}
if (setupMethod != null) {
- builder.setSetup(
+ signatureBuilder.setSetup(
analyzeLifecycleMethod(errors.forMethod(DoFn.Setup.class, setupMethod), setupMethod));
}
if (teardownMethod != null) {
- builder.setTeardown(
+ signatureBuilder.setTeardown(
analyzeLifecycleMethod(
errors.forMethod(DoFn.Teardown.class, teardownMethod), teardownMethod));
}
@@ -205,7 +327,7 @@ public class DoFnSignatures {
if (getInitialRestrictionMethod != null) {
getInitialRestrictionErrors =
errors.forMethod(DoFn.GetInitialRestriction.class, getInitialRestrictionMethod);
- builder.setGetInitialRestriction(
+ signatureBuilder.setGetInitialRestriction(
getInitialRestriction =
analyzeGetInitialRestrictionMethod(
getInitialRestrictionErrors, fnT, getInitialRestrictionMethod, inputT));
@@ -215,7 +337,7 @@ public class DoFnSignatures {
if (splitRestrictionMethod != null) {
ErrorReporter splitRestrictionErrors =
errors.forMethod(DoFn.SplitRestriction.class, splitRestrictionMethod);
- builder.setSplitRestriction(
+ signatureBuilder.setSplitRestriction(
splitRestriction =
analyzeSplitRestrictionMethod(
splitRestrictionErrors, fnT, splitRestrictionMethod, inputT));
@@ -225,7 +347,7 @@ public class DoFnSignatures {
if (getRestrictionCoderMethod != null) {
ErrorReporter getRestrictionCoderErrors =
errors.forMethod(DoFn.GetRestrictionCoder.class, getRestrictionCoderMethod);
- builder.setGetRestrictionCoder(
+ signatureBuilder.setGetRestrictionCoder(
getRestrictionCoder =
analyzeGetRestrictionCoderMethod(
getRestrictionCoderErrors, fnT, getRestrictionCoderMethod));
@@ -234,13 +356,16 @@ public class DoFnSignatures {
DoFnSignature.NewTrackerMethod newTracker = null;
if (newTrackerMethod != null) {
ErrorReporter newTrackerErrors = errors.forMethod(DoFn.NewTracker.class, newTrackerMethod);
- builder.setNewTracker(
+ signatureBuilder.setNewTracker(
newTracker = analyzeNewTrackerMethod(newTrackerErrors, fnT, newTrackerMethod));
}
- builder.setIsBoundedPerElement(inferBoundedness(fnT, processElement, errors));
+ signatureBuilder.setIsBoundedPerElement(inferBoundedness(fnT, processElement, errors));
- DoFnSignature signature = builder.build();
+ signatureBuilder.setStateDeclarations(fnContext.getStateDeclarations());
+ signatureBuilder.setTimerDeclarations(fnContext.getTimerDeclarations());
+
+ DoFnSignature signature = signatureBuilder.build();
// Additional validation for splittable DoFn's.
if (processElement.isSplittable()) {
@@ -452,14 +577,49 @@ public class DoFnSignatures {
}
@VisibleForTesting
+ static DoFnSignature.OnTimerMethod analyzeOnTimerMethod(
+ ErrorReporter errors,
+ TypeDescriptor<? extends DoFn<?, ?>> fnClass,
+ Method m,
+ String timerId,
+ TypeDescriptor<?> outputT,
+ FnAnalysisContext fnContext) {
+ errors.checkArgument(void.class.equals(m.getReturnType()), "Must return void");
+
+ Type[] params = m.getGenericParameterTypes();
+
+ MethodAnalysisContext methodContext = MethodAnalysisContext.create();
+
+ List<DoFnSignature.Parameter> extraParameters = new ArrayList<>();
+ TypeDescriptor<?> expectedOutputReceiverT = outputReceiverTypeOf(outputT);
+ ErrorReporter onTimerErrors = errors.forMethod(DoFn.OnTimer.class, m);
+ for (int i = 0; i < params.length; ++i) {
+ extraParameters.add(
+ analyzeExtraParameter(
+ onTimerErrors,
+ fnContext,
+ methodContext,
+ fnClass,
+ ParameterDescription.of(
+ m,
+ i,
+ fnClass.resolveType(params[i]),
+ Arrays.asList(m.getParameterAnnotations()[i])),
+ null /* restriction type not applicable */,
+ expectedOutputReceiverT));
+ }
+
+ return DoFnSignature.OnTimerMethod.create(m, timerId, extraParameters);
+ }
+
+ @VisibleForTesting
static DoFnSignature.ProcessElementMethod analyzeProcessElementMethod(
ErrorReporter errors,
TypeDescriptor<? extends DoFn<?, ?>> fnClass,
Method m,
TypeDescriptor<?> inputT,
TypeDescriptor<?> outputT,
- Map<String, StateDeclaration> stateDeclarations,
- Map<String, TimerDeclaration> timerDeclarations) {
+ FnAnalysisContext fnContext) {
errors.checkArgument(
void.class.equals(m.getReturnType())
|| DoFn.ProcessContinuation.class.equals(m.getReturnType()),
@@ -468,6 +628,8 @@ public class DoFnSignatures {
TypeDescriptor<?> processContextT = doFnProcessContextTypeOf(inputT, outputT);
+ MethodAnalysisContext methodContext = MethodAnalysisContext.create();
+
Type[] params = m.getGenericParameterTypes();
TypeDescriptor<?> contextT = null;
if (params.length > 0) {
@@ -478,192 +640,211 @@ public class DoFnSignatures {
"Must take %s as the first argument",
formatType(processContextT));
- List<DoFnSignature.Parameter> extraParameters = new ArrayList<>();
- Map<String, DoFnSignature.Parameter> stateParameters = new HashMap<>();
- Map<String, DoFnSignature.Parameter> timerParameters = new HashMap<>();
- TypeDescriptor<?> trackerT = null;
-
+ TypeDescriptor<?> trackerT = getTrackerType(fnClass, m);
TypeDescriptor<?> expectedInputProviderT = inputProviderTypeOf(inputT);
TypeDescriptor<?> expectedOutputReceiverT = outputReceiverTypeOf(outputT);
for (int i = 1; i < params.length; ++i) {
- TypeDescriptor<?> paramT = fnClass.resolveType(params[i]);
- Class<?> rawType = paramT.getRawType();
- if (rawType.equals(BoundedWindow.class)) {
- errors.checkArgument(
- !extraParameters.contains(DoFnSignature.Parameter.boundedWindow()),
- "Multiple %s parameters",
- BoundedWindow.class.getSimpleName());
- extraParameters.add(DoFnSignature.Parameter.boundedWindow());
- } else if (rawType.equals(DoFn.InputProvider.class)) {
- errors.checkArgument(
- !extraParameters.contains(DoFnSignature.Parameter.inputProvider()),
- "Multiple %s parameters",
- DoFn.InputProvider.class.getSimpleName());
- errors.checkArgument(
- paramT.equals(expectedInputProviderT),
- "Wrong type of %s parameter: %s, should be %s",
- DoFn.InputProvider.class.getSimpleName(),
- formatType(paramT),
- formatType(expectedInputProviderT));
- extraParameters.add(DoFnSignature.Parameter.inputProvider());
- } else if (rawType.equals(DoFn.OutputReceiver.class)) {
- errors.checkArgument(
- !extraParameters.contains(DoFnSignature.Parameter.outputReceiver()),
- "Multiple %s parameters",
- DoFn.OutputReceiver.class.getSimpleName());
- errors.checkArgument(
- paramT.equals(expectedOutputReceiverT),
- "Wrong type of %s parameter: %s, should be %s",
- DoFn.OutputReceiver.class.getSimpleName(),
- formatType(paramT),
- formatType(expectedOutputReceiverT));
- extraParameters.add(DoFnSignature.Parameter.outputReceiver());
- } else if (Timer.class.equals(rawType)) {
- // m.getParameters() is not available until Java 8
- Annotation[] annotations = m.getParameterAnnotations()[i];
- String id = null;
- for (Annotation anno : annotations) {
- if (anno.annotationType().equals(DoFn.TimerId.class)) {
- id = ((DoFn.TimerId) anno).value();
- break;
- }
- }
- errors.checkArgument(
- id != null,
- "%s parameter of type %s at index %s missing %s annotation",
- fnClass.getRawType().getName(),
- params[i],
- i,
- DoFn.TimerId.class.getSimpleName());
- errors.checkArgument(
- !timerParameters.containsKey(id),
- "%s parameter of type %s at index %s duplicates %s(\"%s\") on other parameter",
- fnClass.getRawType().getName(),
- params[i],
- i,
- DoFn.TimerId.class.getSimpleName(),
- id);
-
- TimerDeclaration timerDecl = timerDeclarations.get(id);
- errors.checkArgument(
- timerDecl != null,
- "%s parameter of type %s at index %s references undeclared %s \"%s\"",
- fnClass.getRawType().getName(),
- params[i],
- i,
- TimerId.class.getSimpleName(),
- id);
+ Parameter extraParam =
+ analyzeExtraParameter(
+ errors.forMethod(DoFn.ProcessElement.class, m),
+ fnContext,
+ methodContext,
+ fnClass,
+ ParameterDescription.of(
+ m,
+ i,
+ fnClass.resolveType(params[i]),
+ Arrays.asList(m.getParameterAnnotations()[i])),
+ expectedInputProviderT,
+ expectedOutputReceiverT);
+
+ methodContext.addParameter(extraParam);
+ }
- errors.checkArgument(
- timerDecl.field().getDeclaringClass().equals(m.getDeclaringClass()),
- "Method %s has %s parameter at index %s for timer %s"
- + " declared in a different class %s."
- + " Timers may be referenced only in the lexical scope where they are declared.",
- m,
- Timer.class.getSimpleName(),
- i,
- id,
- timerDecl.field().getDeclaringClass().getName());
+ // A splittable DoFn can not have any other extra context parameters.
+ if (methodContext.getExtraParameters().contains(DoFnSignature.Parameter.restrictionTracker())) {
+ errors.checkArgument(
+ methodContext.getExtraParameters().size() == 1,
+ "Splittable DoFn must not have any extra arguments, but has: %s",
+ trackerT,
+ methodContext.getExtraParameters());
+ }
- DoFnSignature.Parameter.TimerParameter timerParameter = Parameter.timerParameter(timerDecl);
- timerParameters.put(id, timerParameter);
- extraParameters.add(timerParameter);
+ return DoFnSignature.ProcessElementMethod.create(
+ m,
+ methodContext.getExtraParameters(),
+ trackerT,
+ DoFn.ProcessContinuation.class.equals(m.getReturnType()));
+ }
- } else if (RestrictionTracker.class.isAssignableFrom(rawType)) {
- errors.checkArgument(
- !extraParameters.contains(DoFnSignature.Parameter.restrictionTracker()),
- "Multiple %s parameters",
- RestrictionTracker.class.getSimpleName());
- extraParameters.add(DoFnSignature.Parameter.restrictionTracker());
- trackerT = paramT;
- } else if (State.class.isAssignableFrom(rawType)) {
- // m.getParameters() is not available until Java 8
- Annotation[] annotations = m.getParameterAnnotations()[i];
- String id = null;
- for (Annotation anno : annotations) {
- if (anno.annotationType().equals(DoFn.StateId.class)) {
- id = ((DoFn.StateId) anno).value();
- break;
- }
- }
- errors.checkArgument(
- id != null,
- "%s parameter of type %s at index %s missing %s annotation",
- fnClass.getRawType().getName(),
- params[i],
- i,
- DoFn.StateId.class.getSimpleName());
+ private static Parameter analyzeExtraParameter(
+ ErrorReporter methodErrors,
+ FnAnalysisContext fnContext,
+ MethodAnalysisContext methodContext,
+ TypeDescriptor<? extends DoFn<?, ?>> fnClass,
+ ParameterDescription param,
+ TypeDescriptor<?> expectedInputProviderT,
+ TypeDescriptor<?> expectedOutputReceiverT) {
+ TypeDescriptor<?> paramT = param.getType();
+ Class<?> rawType = paramT.getRawType();
+
+ ErrorReporter paramErrors = methodErrors.forParameter(param);
+
+ if (rawType.equals(BoundedWindow.class)) {
+ methodErrors.checkArgument(
+ !methodContext.getExtraParameters().contains(Parameter.boundedWindow()),
+ "Multiple %s parameters",
+ BoundedWindow.class.getSimpleName());
+ return Parameter.boundedWindow();
+ } else if (rawType.equals(DoFn.InputProvider.class)) {
+ methodErrors.checkArgument(
+ !methodContext.getExtraParameters().contains(Parameter.inputProvider()),
+ "Multiple %s parameters",
+ DoFn.InputProvider.class.getSimpleName());
+ paramErrors.checkArgument(
+ paramT.equals(expectedInputProviderT),
+ "%s is for %s when it should be %s",
+ DoFn.InputProvider.class.getSimpleName(),
+ formatType(paramT),
+ formatType(expectedInputProviderT));
+ return Parameter.inputProvider();
+ } else if (rawType.equals(DoFn.OutputReceiver.class)) {
+ methodErrors.checkArgument(
+ !methodContext.getExtraParameters().contains(Parameter.outputReceiver()),
+ "Multiple %s parameters",
+ DoFn.OutputReceiver.class.getSimpleName());
+ paramErrors.checkArgument(
+ paramT.equals(expectedOutputReceiverT),
+ "%s is for %s when it should be %s",
+ DoFn.OutputReceiver.class.getSimpleName(),
+ formatType(paramT),
+ formatType(expectedOutputReceiverT));
+ return Parameter.outputReceiver();
+
+ } else if (RestrictionTracker.class.isAssignableFrom(rawType)) {
+ methodErrors.checkArgument(
+ !methodContext.getExtraParameters().contains(Parameter.restrictionTracker()),
+ "Multiple %s parameters",
+ RestrictionTracker.class.getSimpleName());
+ return Parameter.restrictionTracker();
+
+ } else if (rawType.equals(Timer.class)) {
+ // m.getParameters() is not available until Java 8
+ String id = getTimerId(param.getAnnotations());
+
+ paramErrors.checkArgument(
+ id != null,
+ "%s missing %s annotation",
+ Timer.class.getSimpleName(),
+ TimerId.class.getSimpleName());
+
+ paramErrors.checkArgument(
+ !methodContext.getTimerParameters().containsKey(id),
+ "duplicate %s: \"%s\"",
+ TimerId.class.getSimpleName(),
+ id);
- errors.checkArgument(
- !stateParameters.containsKey(id),
- "%s parameter of type %s at index %s duplicates %s(\"%s\") on other parameter",
- fnClass.getRawType().getName(),
- params[i],
- i,
- DoFn.StateId.class.getSimpleName(),
- id);
+ TimerDeclaration timerDecl = fnContext.getTimerDeclarations().get(id);
+ paramErrors.checkArgument(
+ timerDecl != null,
+ "reference to undeclared %s: \"%s\"",
+ TimerId.class.getSimpleName(),
+ id);
- // By static typing this is already a well-formed State subclass
- TypeDescriptor<? extends State> stateType =
- (TypeDescriptor<? extends State>)
- TypeDescriptor.of(fnClass.getType())
- .resolveType(params[i]);
+ paramErrors.checkArgument(
+ timerDecl.field().getDeclaringClass().equals(param.getMethod().getDeclaringClass()),
+ "%s %s declared in a different class %s."
+ + " Timers may be referenced only in the lexical scope where they are declared.",
+ TimerId.class.getSimpleName(),
+ id,
+ timerDecl.field().getDeclaringClass().getName());
+
+ return Parameter.timerParameter(timerDecl);
+
+ } else if (State.class.isAssignableFrom(rawType)) {
+ // m.getParameters() is not available until Java 8
+ String id = getStateId(param.getAnnotations());
+ paramErrors.checkArgument(
+ id != null,
+ "missing %s annotation",
+ DoFn.StateId.class.getSimpleName());
+
+ paramErrors.checkArgument(
+ !methodContext.getStateParameters().containsKey(id),
+ "duplicate %s: \"%s\"",
+ DoFn.StateId.class.getSimpleName(),
+ id);
- StateDeclaration stateDecl = stateDeclarations.get(id);
- errors.checkArgument(
- stateDecl != null,
- "%s parameter of type %s at index %s references undeclared %s \"%s\"",
- fnClass.getRawType().getName(),
- params[i],
- i,
- DoFn.StateId.class.getSimpleName(),
- id);
+ // By static typing this is already a well-formed State subclass
+ TypeDescriptor<? extends State> stateType = (TypeDescriptor<? extends State>) param.getType();
- errors.checkArgument(
- stateDecl.stateType().equals(stateType),
- "%s parameter at index %s has type %s but is a reference to StateId %s of type %s",
- fnClass.getRawType().getName(),
- i,
- params[i],
- id,
- stateDecl.stateType());
+ StateDeclaration stateDecl = fnContext.getStateDeclarations().get(id);
+ paramErrors.checkArgument(
+ stateDecl != null,
+ "reference to undeclared %s: \"%s\"",
+ DoFn.StateId.class.getSimpleName(),
+ id);
- errors.checkArgument(
- stateDecl.field().getDeclaringClass().equals(m.getDeclaringClass()),
- "Method %s has State parameter at index %s for state %s"
- + " declared in a different class %s."
- + " State may be referenced only in the class where it is declared.",
- m,
- i,
- id,
- stateDecl.field().getDeclaringClass().getName());
-
- DoFnSignature.Parameter.StateParameter stateParameter = Parameter.stateParameter(stateDecl);
- stateParameters.put(id, stateParameter);
- extraParameters.add(stateParameter);
- } else {
- List<String> allowedParamTypes =
- Arrays.asList(
- formatType(new TypeDescriptor<BoundedWindow>() {}),
- formatType(new TypeDescriptor<RestrictionTracker<?>>() {}));
- errors.throwIllegalArgument(
- "%s is not a valid context parameter. Should be one of %s",
- formatType(paramT), allowedParamTypes);
+ paramErrors.checkArgument(
+ stateDecl.stateType().equals(stateType),
+ "reference to %s %s with different type %s",
+ StateId.class.getSimpleName(),
+ id,
+ stateDecl.stateType());
+
+ paramErrors.checkArgument(
+ stateDecl.field().getDeclaringClass().equals(param.getMethod().getDeclaringClass()),
+ "%s %s declared in a different class %s."
+ + " State may be referenced only in the class where it is declared.",
+ StateId.class.getSimpleName(),
+ id,
+ stateDecl.field().getDeclaringClass().getName());
+
+ return Parameter.stateParameter(stateDecl);
+ } else {
+ List<String> allowedParamTypes =
+ Arrays.asList(
+ formatType(new TypeDescriptor<BoundedWindow>() {}),
+ formatType(new TypeDescriptor<RestrictionTracker<?>>() {}));
+ paramErrors.throwIllegalArgument(
+ "%s is not a valid context parameter. Should be one of %s",
+ formatType(paramT), allowedParamTypes);
+ // Unreachable
+ return null;
+ }
+ }
+
+ @Nullable
+ private static String getTimerId(List<Annotation> annotations) {
+ for (Annotation anno : annotations) {
+ if (anno.annotationType().equals(DoFn.TimerId.class)) {
+ return ((DoFn.TimerId) anno).value();
}
}
+ return null;
+ }
- // A splittable DoFn can not have any other extra context parameters.
- if (extraParameters.contains(DoFnSignature.Parameter.restrictionTracker())) {
- errors.checkArgument(
- extraParameters.size() == 1,
- "Splittable DoFn must not have any extra arguments apart from BoundedWindow, but has: %s",
- trackerT,
- extraParameters);
+ @Nullable
+ private static String getStateId(List<Annotation> annotations) {
+ for (Annotation anno : annotations) {
+ if (anno.annotationType().equals(DoFn.StateId.class)) {
+ return ((DoFn.StateId) anno).value();
+ }
}
+ return null;
+ }
- return DoFnSignature.ProcessElementMethod.create(
- m, extraParameters, trackerT, DoFn.ProcessContinuation.class.equals(m.getReturnType()));
+ @Nullable
+ private static TypeDescriptor<?> getTrackerType(TypeDescriptor<?> fnClass, Method method) {
+ Type[] params = method.getGenericParameterTypes();
+ for (int i = 0; i < params.length; i++) {
+ TypeDescriptor<?> paramT = fnClass.resolveType(params[i]);
+ if (RestrictionTracker.class.isAssignableFrom(paramT.getRawType())) {
+ return paramT;
+ }
+ }
+ return null;
}
@VisibleForTesting
@@ -905,7 +1086,7 @@ public class DoFnSignatures {
return matches;
}
- private static ImmutableMap<String, DoFnSignature.StateDeclaration> analyzeStateDeclarations(
+ private static Map<String, DoFnSignature.StateDeclaration> analyzeStateDeclarations(
ErrorReporter errors,
Class<?> fnClazz) {
@@ -1015,6 +1196,14 @@ public class DoFnSignatures {
annotation.getSimpleName(), (method == null) ? "(absent)" : format(method)));
}
+ ErrorReporter forParameter(ParameterDescription param) {
+ return new ErrorReporter(
+ this,
+ String.format(
+ "parameter of type %s at index %s",
+ param.getType(), param.getIndex()));
+ }
+
void throwIllegalArgument(String message, Object... args) {
throw new IllegalArgumentException(label + ": " + String.format(message, args));
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8bf6d92c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesProcessElementTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesProcessElementTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesProcessElementTest.java
index 329a099..6cbc95e 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesProcessElementTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesProcessElementTest.java
@@ -96,9 +96,9 @@ public class DoFnSignaturesProcessElementTest {
@Test
public void testBadGenericsTwoArgs() throws Exception {
thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage(
- "Wrong type of OutputReceiver parameter: "
- + "OutputReceiver<Integer>, should be OutputReceiver<String>");
+ thrown.expectMessage("OutputReceiver<Integer>");
+ thrown.expectMessage("should be");
+ thrown.expectMessage("OutputReceiver<String>");
analyzeProcessElementMethod(
new AnonymousMethod() {
@@ -112,9 +112,9 @@ public class DoFnSignaturesProcessElementTest {
@Test
public void testBadGenericWildCards() throws Exception {
thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage(
- "Wrong type of OutputReceiver parameter: "
- + "OutputReceiver<? super Integer>, should be OutputReceiver<String>");
+ thrown.expectMessage("OutputReceiver<? super Integer>");
+ thrown.expectMessage("should be");
+ thrown.expectMessage("OutputReceiver<String>");
analyzeProcessElementMethod(
new AnonymousMethod() {
@@ -137,9 +137,9 @@ public class DoFnSignaturesProcessElementTest {
@Test
public void testBadTypeVariables() throws Exception {
thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage(
- "Wrong type of OutputReceiver parameter: "
- + "OutputReceiver<InputT>, should be OutputReceiver<OutputT>");
+ thrown.expectMessage("OutputReceiver<InputT>");
+ thrown.expectMessage("should be");
+ thrown.expectMessage("OutputReceiver<OutputT>");
DoFnSignatures.INSTANCE.getSignature(BadTypeVariables.class);
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8bf6d92c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
index 573701b..0751b59 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
@@ -88,7 +88,6 @@ public class DoFnSignaturesSplittableDoFnTest {
public void testSplittableProcessElementMustNotHaveOtherParams() throws Exception {
thrown.expect(IllegalArgumentException.class);
thrown.expectMessage("must not have any extra arguments");
- thrown.expectMessage("BoundedWindow");
DoFnSignature.ProcessElementMethod signature =
analyzeProcessElementMethod(
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8bf6d92c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
index 52ecb2a..4187e0a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
@@ -21,6 +21,7 @@ import static org.apache.beam.sdk.transforms.reflect.DoFnSignaturesTestUtils.err
import static org.hamcrest.Matchers.anyOf;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.not;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
@@ -29,11 +30,11 @@ import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFn.OnTimer;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignaturesTestUtils.FakeDoFn;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.TimeDomain;
import org.apache.beam.sdk.util.Timer;
import org.apache.beam.sdk.util.TimerSpec;
@@ -249,7 +250,7 @@ public class DoFnSignaturesTest {
@Test
public void testTimerParameterDuplicate() throws Exception {
thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage("duplicates");
+ thrown.expectMessage("duplicate");
thrown.expectMessage("my-id");
thrown.expectMessage("myProcessElement");
thrown.expectMessage("index 2");
@@ -291,6 +292,28 @@ public class DoFnSignaturesTest {
}
@Test
+ public void testWindowParamOnTimer() throws Exception {
+ final String timerId = "some-timer-id";
+
+ DoFnSignature sig =
+ DoFnSignatures.INSTANCE.getSignature(new DoFn<String, String>() {
+ @TimerId(timerId)
+ private final TimerSpec myfield1 = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+ @ProcessElement
+ public void process(ProcessContext c) {}
+
+ @OnTimer(timerId)
+ public void onTimer(BoundedWindow w) {}
+ }.getClass());
+
+ assertThat(sig.onTimerMethods().get(timerId).extraParameters().size(), equalTo(1));
+ assertThat(
+ sig.onTimerMethods().get(timerId).extraParameters().get(0),
+ instanceOf(DoFnSignature.Parameter.BoundedWindowParameter.class));
+ }
+
+ @Test
public void testDeclAndUsageOfTimerInSuperclass() throws Exception {
DoFnSignature sig =
DoFnSignatures.INSTANCE.getSignature(new DoFnOverridingAbstractTimerUse().getClass());
@@ -525,7 +548,7 @@ public class DoFnSignaturesTest {
@Test
public void testStateParameterDuplicate() throws Exception {
thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage("duplicates");
+ thrown.expectMessage("duplicate");
thrown.expectMessage("my-id");
thrown.expectMessage("myProcessElement");
thrown.expectMessage("index 2");
@@ -549,7 +572,8 @@ public class DoFnSignaturesTest {
public void testStateParameterWrongStateType() throws Exception {
thrown.expect(IllegalArgumentException.class);
thrown.expectMessage("WatermarkHoldState");
- thrown.expectMessage("but is a reference to");
+ thrown.expectMessage("reference to");
+ thrown.expectMessage("different type");
thrown.expectMessage("ValueState");
thrown.expectMessage("my-id");
thrown.expectMessage("myProcessElement");
@@ -572,7 +596,8 @@ public class DoFnSignaturesTest {
public void testStateParameterWrongGenericType() throws Exception {
thrown.expect(IllegalArgumentException.class);
thrown.expectMessage("ValueState<java.lang.String>");
- thrown.expectMessage("but is a reference to");
+ thrown.expectMessage("reference to");
+ thrown.expectMessage("different type");
thrown.expectMessage("ValueState<java.lang.Integer>");
thrown.expectMessage("my-id");
thrown.expectMessage("myProcessElement");
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8bf6d92c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java
index 49e2ba7..b7d137a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java
@@ -18,9 +18,9 @@
package org.apache.beam.sdk.transforms.reflect;
import java.lang.reflect.Method;
-import java.util.Collections;
import java.util.NoSuchElementException;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignatures.FnAnalysisContext;
import org.apache.beam.sdk.values.TypeDescriptor;
/** Utilities for use in {@link DoFnSignatures} tests. */
@@ -61,7 +61,6 @@ class DoFnSignaturesTestUtils {
method.getMethod(),
TypeDescriptor.of(Integer.class),
TypeDescriptor.of(String.class),
- Collections.EMPTY_MAP,
- Collections.EMPTY_MAP);
+ FnAnalysisContext.create());
}
}
[44/50] incubator-beam git commit: Added Regex Transform and test.
Posted by th...@apache.org.
Added Regex Transform and test.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bc28799d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bc28799d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bc28799d
Branch: refs/heads/apex-runner
Commit: bc28799d575341d4ab359c971a10514f518249a0
Parents: 70255d2
Author: Jesse Anderson <je...@smokinghand.com>
Authored: Mon Jun 20 16:44:43 2016 -0700
Committer: Jesse Anderson <je...@smokinghand.com>
Committed: Mon Nov 7 09:39:29 2016 -0800
----------------------------------------------------------------------
.../beam/sdk/transforms/RegexTransform.java | 505 +++++++++++++++++++
.../beam/sdk/transforms/RegexTransformTest.java | 262 ++++++++++
2 files changed, 767 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bc28799d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/RegexTransform.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/RegexTransform.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/RegexTransform.java
new file mode 100644
index 0000000..bd7848a
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/RegexTransform.java
@@ -0,0 +1,505 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+
+/**
+ * {@code PTransorm}s to use Regular Expressions to process elements in a
+ * {@link PCollection}.
+ *
+ * <p>
+ * {@link RegexTransform#matches(String, int)} can be used to see if an entire line matches
+ * a Regex. {@link RegexTransform#matchesKV(String, int, int)} can be used to see if an entire
+ * line matches a Regex and output certain groups as a {@link KV}.
+ * </p>
+ * <p>
+ * {@link RegexTransform#find(String, int)} can be used to see if a portion of a line
+ * matches a Regex. {@link RegexTransform#matchesKV(String, int, int)} can be used to see if a
+ * portion of a line matches a Regex and output certain groups as a {@link KV}.
+ * </p>
+ * <p>
+ * Lines that do not match the Regex will not be output.
+ * </p>
+ */
+public class RegexTransform {
+ private RegexTransform() {
+ // do not instantiate
+ }
+
+ /**
+ * Returns a {@link RegexTransform.Matches} {@link PTransform} that checks if
+ * the entire line matches the Regex. Returns the entire line (group 0) as a
+ * {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ */
+ public static Matches matches(String regex) {
+ return matches(regex, 0);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.Matches} {@link PTransform} that checks if
+ * the entire line matches the Regex. Returns the group as a
+ * {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param group
+ * The Regex group to return as a PCollection
+ */
+ public static Matches matches(String regex, int group) {
+ return new Matches(regex, group);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.MatchesKV} {@link PTransform} that checks
+ * if the entire line matches the Regex. Returns the specified groups as the
+ * key and value as a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param keyGroup
+ * The Regex group to use as the key
+ * @param valueGroup
+ * The Regex group to use the value
+ */
+ public static MatchesKV matchesKV(String regex, int keyGroup,
+ int valueGroup) {
+ return new MatchesKV(regex, keyGroup, valueGroup);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.Find} {@link PTransform} that checks if a
+ * portion of the line matches the Regex. Returns the entire line (group 0) as
+ * a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ */
+ public static Find find(String regex) {
+ return find(regex, 0);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.Find} {@link PTransform} that checks if a
+ * portion of the line matches the Regex. Returns the group as a
+ * {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param group
+ * The Regex group to return as a PCollection
+ */
+ public static Find find(String regex, int group) {
+ return new Find(regex, group);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.FindKV} {@link PTransform} that checks if a
+ * portion of the line matches the Regex. Returns the specified groups as the
+ * key and value as a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param keyGroup
+ * The Regex group to use as the key
+ * @param valueGroup
+ * The Regex group to use the value
+ */
+ public static FindKV findKV(String regex, int keyGroup, int valueGroup) {
+ return new FindKV(regex, keyGroup, valueGroup);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.ReplaceAll} {@link PTransform} that checks if a
+ * portion of the line matches the Regex and replaces all matches with the replacement
+ * String. Returns the group as a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param replacement
+ * The string to be substituted for each match
+ */
+ public static ReplaceAll replaceAll(String regex, String replacement) {
+ return new ReplaceAll(regex, replacement);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.ReplaceAll} {@link PTransform} that checks if a
+ * portion of the line matches the Regex and replaces the first match with the replacement
+ * String. Returns the group as a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param replacement
+ * The string to be substituted for each match
+ */
+ public static ReplaceFirst replaceFirst(String regex, String replacement) {
+ return new ReplaceFirst(regex, replacement);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.Split} {@link PTransform} that splits a string
+ * on the regular expression and then outputs each item. It will not output empty
+ * items. Returns the group as a {@link PCollection}.
+ * a {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ */
+ public static Split split(String regex) {
+ return split(regex, false);
+ }
+
+ /**
+ * Returns a {@link RegexTransform.Split} {@link PTransform} that splits a string
+ * on the regular expression and then outputs each item. Returns the group as a
+ * {@link PCollection}.
+ * @param regex
+ * The regular expression to run
+ * @param outputEmpty
+ * Should empty be output. True to output empties and false if not.
+ */
+ public static Split split(String regex, boolean outputEmpty) {
+ return new Split(regex, outputEmpty);
+ }
+
+ /**
+ * {@code RegexTransform.Matches<String>} takes a {@code PCollection<String>}
+ * and returns a {@code PCollection<String>} representing the value
+ * extracted from the Regex groups of the input {@code PCollection}
+ * to the number of times that element occurs in the input.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If the entire line
+ * does not match the Regex, the line will not be output. If it does match the
+ * entire line, the group in the Regex will be used. The output will be the
+ * Regex group.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(RegexTransform.matches("myregex (mygroup)", 1));
+ * }
+ * </pre>
+ */
+ public static class Matches
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ int group;
+
+ public Matches(String regex, int group) {
+ this.pattern = Pattern.compile(regex);
+ this.group = group;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in
+ .apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.matches()) {
+ c.output(m.group(group));
+ }
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code RegexTransform.MatchesKV<KV<String, String>>} takes a
+ * {@code PCollection<String>} and returns a
+ * {@code PCollection<KV<String, String>>} representing the key and value
+ * extracted from the Regex groups of the input {@code PCollection} to the
+ * number of times that element occurs in the input.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If the entire line
+ * does not match the Regex, the line will not be output. If it does match the
+ * entire line, the groups in the Regex will be used. The key will be the
+ * key's group and the value will be the value's group.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<KV<String, String>> keysAndValues =
+ * words.apply(RegexTransform.matchesKV("myregex (mykeygroup) (myvaluegroup)", 1, 2));
+ * }
+ * </pre>
+ */
+ public static class MatchesKV
+ extends PTransform<PCollection<String>, PCollection<KV<String, String>>> {
+ Pattern pattern;
+ int keyGroup, valueGroup;
+
+ public MatchesKV(String regex, int keyGroup, int valueGroup) {
+ this.pattern = Pattern.compile(regex);
+ this.keyGroup = keyGroup;
+ this.valueGroup = valueGroup;
+ }
+
+ public PCollection<KV<String, String>> apply(PCollection<String> in) {
+ return in.apply(ParDo
+ .of(new DoFn<String, KV<String, String>>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
+ }
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code RegexTransform.Find<String>} takes a {@code PCollection<String>} and
+ * returns a {@code PCollection<String>} representing the value extracted
+ * from the Regex groups of the input {@code PCollection} to
+ * the number of times that element occurs in the input.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If a portion of the
+ * line does not match the Regex, the line will not be output. If it does
+ * match a portion of the line, the group in the Regex will be used. The
+ * output will be the Regex group.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(RegexTransform.find("myregex (mygroup)", 1));
+ * }
+ * </pre>
+ */
+ public static class Find
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ int group;
+
+ public Find(String regex, int group) {
+ this.pattern = Pattern.compile(regex);
+ this.group = group;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in.apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(m.group(group));
+ }
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code RegexTransform.MatchesKV<KV<String, String>>} takes a
+ * {@code PCollection<String>} and returns a
+ * {@code PCollection<KV<String, String>>} representing the key and value
+ * extracted from the Regex groups of the input {@code PCollection} to the
+ * number of times that element occurs in the input.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If a portion of the
+ * line does not match the Regex, the line will not be output. If it does
+ * match a portion of the line, the groups in the Regex will be used. The key
+ * will be the key's group and the value will be the value's group.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<KV<String, String>> keysAndValues =
+ * words.apply(RegexTransform.findKV("myregex (mykeygroup) (myvaluegroup)", 1, 2));
+ * }
+ * </pre>
+ */
+ public static class FindKV
+ extends PTransform<PCollection<String>, PCollection<KV<String, String>>> {
+ Pattern pattern;
+ int keyGroup, valueGroup;
+
+ public FindKV(String regex, int keyGroup, int valueGroup) {
+ this.pattern = Pattern.compile(regex);
+ this.keyGroup = keyGroup;
+ this.valueGroup = valueGroup;
+ }
+
+ public PCollection<KV<String, String>> apply(PCollection<String> in) {
+ return in.apply(
+ ParDo.of(new DoFn<String, KV<String, String>>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
+ }
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code RegexTransform.ReplaceAll<String>} takes a {@code PCollection<String>} and
+ * returns a {@code PCollection<String>} with all Strings that matched the
+ * Regex being replaced with the replacement string.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If a portion of the
+ * line does not match the Regex, the line will be output without changes. If it does
+ * match a portion of the line, all portions matching the Regex will be replaced
+ * with the replacement String.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(RegexTransform.replaceAll("myregex", "myreplacement"));
+ * }
+ * </pre>
+ */
+ public static class ReplaceAll
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ String replacement;
+
+ public ReplaceAll(String regex, String replacement) {
+ this.pattern = Pattern.compile(regex);
+ this.replacement = replacement;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in.apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+ c.output(m.replaceAll(replacement));
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code RegexTransform.ReplaceFirst<String>} takes a {@code PCollection<String>} and
+ * returns a {@code PCollection<String>} with the first Strings that matched the
+ * Regex being replaced with the replacement string.
+ *
+ * <p>
+ * This transform runs a Regex on the entire input line. If a portion of the
+ * line does not match the Regex, the line will be output without changes. If it does
+ * match a portion of the line, the first portion matching the Regex will be replaced
+ * with the replacement String.
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(RegexTransform.replaceFirst("myregex", "myreplacement"));
+ * }
+ * </pre>
+ */
+ public static class ReplaceFirst
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ String replacement;
+
+ public ReplaceFirst(String regex, String replacement) {
+ this.pattern = Pattern.compile(regex);
+ this.replacement = replacement;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in.apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+ c.output(m.replaceFirst(replacement));
+ }
+ }));
+ }
+ }
+
+ /**
+ * {@code RegexTransform.Split<String>} takes a {@code PCollection<String>} and
+ * returns a {@code PCollection<String>} with the input string split into
+ * individual items in a list. Each item is then output as a separate string.
+ *
+ * <p>
+ * This transform runs a Regex as part of a splint the entire input line. The split
+ * gives back an array of items. Each item is output as a separate item in the
+ * {@code PCollection<String>}.
+ * </p>
+ *
+ * <p>
+ * Depending on the Regex, a split can be an empty or
+ * "" string. You can pass in a parameter if you want empty strings or not.
+ * </p>
+ *
+ * <p>
+ * Example of use:
+ * <pre>
+ * {@code
+ * PCollection<String> words = ...;
+ * PCollection<String> values =
+ * words.apply(RegexTransform.split("\W*"));
+ * }
+ * </pre>
+ */
+ public static class Split
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ Pattern pattern;
+ boolean outputEmpty;
+
+ public Split(String regex, boolean outputEmpty) {
+ this.pattern = Pattern.compile(regex);
+ this.outputEmpty = outputEmpty;
+ }
+
+ public PCollection<String> apply(PCollection<String> in) {
+ return in.apply(ParDo.of(new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ String[] items = pattern.split(c.element());
+
+ for (String item : items) {
+ if (outputEmpty || !item.isEmpty()) {
+ c.output(item);
+ }
+ }
+ }
+ }));
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bc28799d/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTransformTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTransformTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTransformTest.java
new file mode 100644
index 0000000..63d02d7
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTransformTest.java
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import java.io.Serializable;
+
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link RegexTransform}.
+ */
+@RunWith(JUnit4.class)
+public class RegexTransformTest implements Serializable {
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFind() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("aj", "xj", "yj", "zj"))
+ .apply(RegexTransform.find("[xyz]"));
+
+ PAssert.that(output).containsInAnyOrder("x", "y", "z");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFindGroup() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("aj", "xj", "yj", "zj"))
+ .apply(RegexTransform.find("([xyz])", 1));
+
+ PAssert.that(output).containsInAnyOrder("x", "y", "z");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFindNone() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("a", "b", "c", "d"))
+ .apply(RegexTransform.find("[xyz]"));
+
+ PAssert.that(output).empty();
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testKVFind() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<KV<String, String>> output = p
+ .apply(Create.of("a b c"))
+ .apply(RegexTransform.findKV("a (b) (c)", 1, 2));
+
+ PAssert.that(output).containsInAnyOrder(KV.of("b", "c"));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testKVFindNone() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<KV<String, String>> output = p
+ .apply(Create.of("x y z"))
+ .apply(RegexTransform.findKV("a (b) (c)", 1, 2));
+
+ PAssert.that(output).empty();
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMatches() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("a", "x", "y", "z"))
+ .apply(RegexTransform.matches("[xyz]"));
+
+ PAssert.that(output).containsInAnyOrder("x", "y", "z");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMatchesNone() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("a", "b", "c", "d"))
+ .apply(RegexTransform.matches("[xyz]"));
+
+ PAssert.that(output).empty();
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMatchesGroup() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("a", "x xxx", "x yyy", "x zzz"))
+ .apply(RegexTransform.matches("x ([xyz]*)", 1));
+
+ PAssert.that(output).containsInAnyOrder("xxx", "yyy", "zzz");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testKVMatches() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<KV<String, String>> output = p
+ .apply(Create.of("a b c"))
+ .apply(RegexTransform.matchesKV("a (b) (c)", 1, 2));
+
+ PAssert.that(output).containsInAnyOrder(KV.of("b", "c"));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testKVMatchesNone() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<KV<String, String>> output = p
+ .apply(Create.of("x y z"))
+ .apply(RegexTransform.matchesKV("a (b) (c)", 1, 2));
+ PAssert.that(output).empty();
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testReplaceAll() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("xj", "yj", "zj"))
+ .apply(RegexTransform.replaceAll("[xyz]", "new"));
+
+ PAssert.that(output).containsInAnyOrder("newj", "newj", "newj");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testReplaceAllMixed() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("abc", "xj", "yj", "zj", "def"))
+ .apply(RegexTransform.replaceAll("[xyz]", "new"));
+
+ PAssert.that(output).containsInAnyOrder("abc", "newj", "newj", "newj", "def");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testReplaceFirst() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("xjx", "yjy", "zjz"))
+ .apply(RegexTransform.replaceFirst("[xyz]", "new"));
+
+ PAssert.that(output).containsInAnyOrder("newjx", "newjy", "newjz");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testReplaceFirstMixed() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("abc", "xjx", "yjy", "zjz", "def"))
+ .apply(RegexTransform.replaceFirst("[xyz]", "new"));
+
+ PAssert.that(output).containsInAnyOrder("abc", "newjx", "newjy", "newjz", "def");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSplits() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(RegexTransform.split("\\W+"));
+
+ PAssert.that(output).containsInAnyOrder("The", "quick", "brown",
+ "fox", "jumps", "over", "the", "lazy", "dog");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSplitsWithEmpty() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(RegexTransform.split("\\s", true));
+
+ String[] outputStr = "The quick brown fox jumps over the lazy dog".split("\\s");
+
+ PAssert.that(output).containsInAnyOrder("The", "", "quick", "brown", "", "",
+ "fox", "jumps", "over", "", "", "", "the", "lazy", "dog");
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSplitsWithoutEmpty() {
+ TestPipeline p = TestPipeline.create();
+
+ PCollection<String> output = p
+ .apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(RegexTransform.split("\\s", false));
+
+ PAssert.that(output).containsInAnyOrder("The", "quick", "brown",
+ "fox", "jumps", "over", "the", "lazy", "dog");
+ p.run();
+ }
+}
[12/50] incubator-beam git commit: gcp-io: add ApiSurfaceTest
Posted by th...@apache.org.
gcp-io: add ApiSurfaceTest
Copied from SDK, but I manually recreated the whitelist.
Surely there are refactoring improvements to be had so that we can more easily
reuse ApiSurface across modules.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d6f3a02d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d6f3a02d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d6f3a02d
Branch: refs/heads/apex-runner
Commit: d6f3a02d07257ceb30f294fb845e1fed4100362b
Parents: bdd71c4
Author: Dan Halperin <dh...@google.com>
Authored: Mon Oct 24 20:08:20 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Thu Nov 3 12:31:47 2016 -0700
----------------------------------------------------------------------
.../apache/beam/sdk/io/gcp/ApiSurfaceTest.java | 132 +++++++++++++++++++
1 file changed, 132 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d6f3a02d/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/ApiSurfaceTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/ApiSurfaceTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/ApiSurfaceTest.java
new file mode 100644
index 0000000..1f7b292
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/ApiSurfaceTest.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp;
+
+import static org.hamcrest.Matchers.anyOf;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.fail;
+
+import com.google.cloud.bigtable.grpc.BigtableInstanceName;
+import com.google.cloud.bigtable.grpc.BigtableTableName;
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.beam.sdk.util.ApiSurface;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeDiagnosingMatcher;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for the API surface of the gcp-io module. Tests that our public API is conformant to a
+ * hard-coded policy.
+ */
+@RunWith(JUnit4.class)
+public class ApiSurfaceTest {
+
+ @Test
+ public void testOurApiSurface() throws Exception {
+ ApiSurface checkedApiSurface =
+ ApiSurface.ofPackage("org.apache.beam.sdk.io.gcp")
+ .pruningPattern("org[.]apache[.]beam[.].*Test.*")
+ .pruningPattern("org[.]apache[.]beam[.].*IT")
+ .pruningPattern("java[.]lang.*");
+
+ Map<Class<?>, List<Class<?>>> disallowedClasses = Maps.newHashMap();
+ for (Class<?> clazz : checkedApiSurface.getExposedClasses()) {
+ if (!classIsAllowed(clazz)) {
+ disallowedClasses.put(clazz, checkedApiSurface.getAnyExposurePath(clazz));
+ }
+ }
+
+ List<String> disallowedMessages = Lists.newArrayList();
+ for (Map.Entry<Class<?>, List<Class<?>>> entry : disallowedClasses.entrySet()) {
+ disallowedMessages.add(entry.getKey() + " exposed via:\n\t\t"
+ + Joiner.on("\n\t\t").join(entry.getValue()));
+ }
+ Collections.sort(disallowedMessages);
+
+ if (!disallowedMessages.isEmpty()) {
+ fail("The following disallowed classes appear in the public API surface of the SDK:\n\t"
+ + Joiner.on("\n\t").join(disallowedMessages));
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ private static final Set<Matcher<? extends Class<?>>> ALLOWED_PACKAGES =
+ ImmutableSet.<Matcher<? extends Class<?>>>of(
+ inPackage("com.google.api.client.json"),
+ inPackage("com.google.api.client.util"),
+ inPackage("com.google.api.services.bigquery.model"),
+ inPackage("com.google.auth"),
+ inPackage("com.google.bigtable.v2"),
+ inPackage("com.google.cloud.bigtable.config"),
+ equalTo(BigtableInstanceName.class),
+ equalTo(BigtableTableName.class),
+ // https://github.com/GoogleCloudPlatform/cloud-bigtable-client/pull/1056
+ inPackage("com.google.common.collect"), // via Bigtable, PR above out to fix.
+ inPackage("com.google.datastore.v1"),
+ inPackage("com.google.protobuf"),
+ inPackage("com.google.type"),
+ inPackage("com.fasterxml.jackson.annotation"),
+ inPackage("io.grpc"),
+ inPackage("java"),
+ inPackage("javax"),
+ inPackage("org.apache.beam"),
+ inPackage("org.apache.commons.logging"), // via Bigtable
+ inPackage("org.joda.time"));
+
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ private boolean classIsAllowed(Class<?> clazz) {
+ // Safe cast inexpressible in Java without rawtypes
+ return anyOf((Iterable) ALLOWED_PACKAGES).matches(clazz);
+ }
+
+ private static Matcher<Class<?>> inPackage(String packageName) {
+ return new ClassInPackage(packageName);
+ }
+
+ private static class ClassInPackage extends TypeSafeDiagnosingMatcher<Class<?>> {
+
+ private final String packageName;
+
+ public ClassInPackage(String packageName) {
+ this.packageName = packageName;
+ }
+
+ @Override
+ public void describeTo(Description description) {
+ description.appendText("Class in package \"");
+ description.appendText(packageName);
+ description.appendText("\"");
+ }
+
+ @Override
+ protected boolean matchesSafely(Class<?> clazz, Description mismatchDescription) {
+ return clazz.getName().startsWith(packageName + ".");
+ }
+
+ }
+}
[37/50] incubator-beam git commit: [BEAM-889] Let Spark handle the
user-provided checkpointDir, but warn if not a reliable fs.
Posted by th...@apache.org.
[BEAM-889] Let Spark handle the user-provided checkpointDir, but warn if not a reliable fs.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/90a75d1f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/90a75d1f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/90a75d1f
Branch: refs/heads/apex-runner
Commit: 90a75d1fb0706ec4cc25a9eeeca8ade1b3b7de28
Parents: 46fbfe0
Author: Sela <an...@paypal.com>
Authored: Thu Nov 3 18:22:20 2016 +0200
Committer: Sela <an...@paypal.com>
Committed: Fri Nov 4 23:59:40 2016 +0200
----------------------------------------------------------------------
.../runners/spark/SparkPipelineOptions.java | 3 +--
.../SparkRunnerStreamingContextFactory.java | 23 +++++---------------
.../streaming/EmptyStreamAssertionTest.java | 3 +--
.../streaming/FlattenStreamingTest.java | 6 ++---
.../streaming/KafkaStreamingTest.java | 6 ++---
.../ResumeFromCheckpointStreamingTest.java | 3 +--
.../streaming/SimpleStreamingWordCountTest.java | 3 +--
.../utils/TestOptionsForStreaming.java | 12 +++++-----
8 files changed, 19 insertions(+), 40 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a75d1f/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
----------------------------------------------------------------------
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
index 08e14fe..4eada35 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
@@ -77,8 +77,7 @@ public interface SparkPipelineOptions extends PipelineOptions, StreamingOptions,
class TmpCheckpointDirFactory implements DefaultValueFactory<String> {
@Override
public String create(PipelineOptions options) {
- SparkPipelineOptions sparkPipelineOptions = options.as(SparkPipelineOptions.class);
- return "file:///tmp/" + sparkPipelineOptions.getJobName();
+ return "/tmp/" + options.as(SparkPipelineOptions.class).getJobName();
}
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a75d1f/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java
----------------------------------------------------------------------
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java
index 2378788..a670f61 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java
@@ -20,11 +20,6 @@ package org.apache.beam.runners.spark.translation.streaming;
import static com.google.common.base.Preconditions.checkArgument;
-import com.google.common.base.Predicates;
-import com.google.common.collect.Iterables;
-import java.net.MalformedURLException;
-import java.net.URL;
-import java.util.Arrays;
import org.apache.beam.runners.spark.SparkPipelineOptions;
import org.apache.beam.runners.spark.SparkRunner;
import org.apache.beam.runners.spark.translation.SparkContextFactory;
@@ -48,7 +43,7 @@ import org.slf4j.LoggerFactory;
public class SparkRunnerStreamingContextFactory implements JavaStreamingContextFactory {
private static final Logger LOG =
LoggerFactory.getLogger(SparkRunnerStreamingContextFactory.class);
- private static final Iterable<String> KNOWN_RELIABLE_FS = Arrays.asList("hdfs", "s3", "gs");
+ private static final String KNOWN_RELIABLE_FS_PATTERN = "^(hdfs|s3|gs)";
private final Pipeline pipeline;
private final SparkPipelineOptions options;
@@ -83,19 +78,11 @@ public class SparkRunnerStreamingContextFactory implements JavaStreamingContextF
// set checkpoint dir.
String checkpointDir = options.getCheckpointDir();
- LOG.info("Checkpoint dir set to: {}", checkpointDir);
- try {
- // validate checkpoint dir and warn if not of a known durable filesystem.
- URL checkpointDirUrl = new URL(checkpointDir);
- if (!Iterables.any(KNOWN_RELIABLE_FS, Predicates.equalTo(checkpointDirUrl.getProtocol()))) {
- LOG.warn("Checkpoint dir URL {} does not match a reliable filesystem, in case of failures "
- + "this job may not recover properly or even at all.", checkpointDirUrl);
- }
- } catch (MalformedURLException e) {
- throw new RuntimeException("Failed to form checkpoint dir URL. CheckpointDir should be in "
- + "the form of hdfs:///path/to/dir or other reliable fs protocol, "
- + "or file:///path/to/dir for local mode.", e);
+ if (!checkpointDir.matches(KNOWN_RELIABLE_FS_PATTERN)) {
+ LOG.warn("The specified checkpoint dir {} does not match a reliable filesystem so in case "
+ + "of failures this job may not recover properly or even at all.", checkpointDir);
}
+ LOG.info("Checkpoint dir set to: {}", checkpointDir);
jssc.checkpoint(checkpointDir);
// register listeners.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a75d1f/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/EmptyStreamAssertionTest.java
----------------------------------------------------------------------
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/EmptyStreamAssertionTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/EmptyStreamAssertionTest.java
index 4f2a7c6..3e95b4d 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/EmptyStreamAssertionTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/EmptyStreamAssertionTest.java
@@ -55,8 +55,7 @@ public class EmptyStreamAssertionTest implements Serializable {
@Test
public void testAssertion() throws Exception {
- SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(
- checkpointParentDir.newFolder(getClass().getSimpleName()));
+ SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(checkpointParentDir);
Duration windowDuration = new Duration(options.getBatchIntervalMillis());
Pipeline pipeline = Pipeline.create(options);
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a75d1f/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
----------------------------------------------------------------------
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
index e6872f1..319b5e9 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
@@ -61,8 +61,7 @@ public class FlattenStreamingTest {
@Test
public void testFlattenUnbounded() throws Exception {
- SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(
- checkpointParentDir.newFolder(getClass().getSimpleName()));
+ SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(checkpointParentDir);
Pipeline p = Pipeline.create(options);
PCollection<String> w1 =
@@ -81,8 +80,7 @@ public class FlattenStreamingTest {
@Test
public void testFlattenBoundedUnbounded() throws Exception {
- SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(
- checkpointParentDir.newFolder(getClass().getSimpleName()));
+ SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(checkpointParentDir);
Pipeline p = Pipeline.create(options);
PCollection<String> w1 =
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a75d1f/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
----------------------------------------------------------------------
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
index fe2d04e..f01059f 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
@@ -74,8 +74,7 @@ public class KafkaStreamingTest {
@Test
public void testEarliest2Topics() throws Exception {
- SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(
- checkpointParentDir.newFolder(getClass().getSimpleName()));
+ SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(checkpointParentDir);
// It seems that the consumer's first "position" lookup (in unit test) takes +200 msec,
// so to be on the safe side we'll set to 750 msec.
options.setMinReadTimeMillis(750L);
@@ -122,8 +121,7 @@ public class KafkaStreamingTest {
@Test
public void testLatest() throws Exception {
- SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(
- checkpointParentDir.newFolder(getClass().getSimpleName()));
+ SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(checkpointParentDir);
//--- setup
final String topic = "topic";
// messages.
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a75d1f/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
----------------------------------------------------------------------
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
index ca0b668..34ffbe2 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
@@ -108,8 +108,7 @@ public class ResumeFromCheckpointStreamingTest {
@Test
public void testRun() throws Exception {
- SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(
- checkpointParentDir.newFolder(getClass().getSimpleName()));
+ SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(checkpointParentDir);
// It seems that the consumer's first "position" lookup (in unit test) takes +200 msec,
// so to be on the safe side we'll set to 750 msec.
options.setMinReadTimeMillis(750L);
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a75d1f/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SimpleStreamingWordCountTest.java
----------------------------------------------------------------------
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SimpleStreamingWordCountTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SimpleStreamingWordCountTest.java
index 4bc9a3d..edba507 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SimpleStreamingWordCountTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SimpleStreamingWordCountTest.java
@@ -62,8 +62,7 @@ public class SimpleStreamingWordCountTest implements Serializable {
@Test
public void testFixedWindows() throws Exception {
- SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(
- checkpointParentDir.newFolder(getClass().getSimpleName()));
+ SparkPipelineOptions options = commonOptions.withTmpCheckpointDir(checkpointParentDir);
// override defaults
options.setBatchIntervalMillis(BATCH_INTERVAL.getMillis());
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a75d1f/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/utils/TestOptionsForStreaming.java
----------------------------------------------------------------------
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/utils/TestOptionsForStreaming.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/utils/TestOptionsForStreaming.java
index d695df0..2861d9f 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/utils/TestOptionsForStreaming.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/utils/TestOptionsForStreaming.java
@@ -19,12 +19,13 @@
package org.apache.beam.runners.spark.translation.streaming.utils;
-import java.io.File;
-import java.net.MalformedURLException;
+import java.io.IOException;
import org.apache.beam.runners.spark.SparkPipelineOptions;
import org.apache.beam.runners.spark.SparkRunner;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.junit.rules.ExternalResource;
+import org.junit.rules.TemporaryFolder;
+
/**
@@ -41,11 +42,10 @@ public class TestOptionsForStreaming extends ExternalResource {
options.setTimeout(1000L);
}
- public SparkPipelineOptions withTmpCheckpointDir(File checkpointDir)
- throws MalformedURLException {
+ public SparkPipelineOptions withTmpCheckpointDir(TemporaryFolder parent)
+ throws IOException {
// tests use JUnit's TemporaryFolder path in the form of: /.../junit/...
- // so need to add the missing protocol.
- options.setCheckpointDir(checkpointDir.toURI().toURL().toString());
+ options.setCheckpointDir(parent.newFolder(options.getJobName()).toURI().toURL().toString());
return options;
}
[11/50] incubator-beam git commit: [BEAM-790] Validate
PipelineOptions Default annotation.
Posted by th...@apache.org.
[BEAM-790] Validate PipelineOptions Default annotation.
This closes #1159
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bdd71c4b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bdd71c4b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bdd71c4b
Branch: refs/heads/apex-runner
Commit: bdd71c4bcb6d761c5a71b433e2a27379a10da45d
Parents: 6a05cf4 f605b02
Author: Luke Cwik <lc...@google.com>
Authored: Thu Nov 3 11:56:22 2016 -0700
Committer: Luke Cwik <lc...@google.com>
Committed: Thu Nov 3 11:56:22 2016 -0700
----------------------------------------------------------------------
.../sdk/options/PipelineOptionsFactory.java | 248 ++++++++++++++-----
.../sdk/options/ProxyInvocationHandler.java | 5 +-
.../beam/sdk/util/common/ReflectHelpers.java | 17 ++
.../sdk/options/PipelineOptionsFactoryTest.java | 239 ++++++++++++++++++
.../sdk/options/ProxyInvocationHandlerTest.java | 29 ++-
.../sdk/util/common/ReflectHelpersTest.java | 25 ++
6 files changed, 491 insertions(+), 72 deletions(-)
----------------------------------------------------------------------
[35/50] incubator-beam git commit: Remove @Default from IT options
Posted by th...@apache.org.
Remove @Default from IT options
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1eccd29b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1eccd29b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1eccd29b
Branch: refs/heads/apex-runner
Commit: 1eccd29b72aedc87e69763fbcc828b5694335e1f
Parents: 99062d1
Author: Mark Liu <ma...@markliu0.mtv.corp.google.com>
Authored: Thu Nov 3 16:45:43 2016 -0700
Committer: Luke Cwik <lc...@google.com>
Committed: Fri Nov 4 14:19:43 2016 -0700
----------------------------------------------------------------------
.../beam/examples/WindowedWordCountIT.java | 11 ++++++++---
.../org/apache/beam/examples/WordCountIT.java | 19 +++++++++++--------
.../examples/cookbook/BigQueryTornadoesIT.java | 11 ++++++++---
3 files changed, 27 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1eccd29b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
----------------------------------------------------------------------
diff --git a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
index 6742654..d545ad2 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.examples;
+import com.google.common.base.Strings;
import java.io.IOException;
import org.apache.beam.examples.WindowedWordCount.Options;
-import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.StreamingOptions;
import org.apache.beam.sdk.testing.BigqueryMatcher;
@@ -37,12 +37,13 @@ import org.junit.runners.JUnit4;
@RunWith(JUnit4.class)
public class WindowedWordCountIT {
+ private static final String DEFAULT_OUTPUT_CHECKSUM = "ff54f6f42b2afeb146206c1e8e915deaee0362b4";
+
/**
* Options for the {@link WindowedWordCount} Integration Test.
*/
public interface WindowedWordCountITOptions
extends Options, TestPipelineOptions, StreamingOptions {
- @Default.String("ff54f6f42b2afeb146206c1e8e915deaee0362b4")
String getChecksum();
void setChecksum(String value);
}
@@ -66,9 +67,13 @@ public class WindowedWordCountIT {
String query = String.format("SELECT word, SUM(count) FROM [%s:%s.%s] GROUP BY word",
options.getProject(), options.getBigQueryDataset(), options.getBigQueryTable());
+ String outputChecksum =
+ Strings.isNullOrEmpty(options.getChecksum())
+ ? DEFAULT_OUTPUT_CHECKSUM
+ : options.getChecksum();
options.setOnSuccessMatcher(
new BigqueryMatcher(
- options.getAppName(), options.getProject(), query, options.getChecksum()));
+ options.getAppName(), options.getProject(), query, outputChecksum));
WindowedWordCount.main(TestPipeline.convertToArgs(options));
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1eccd29b/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
----------------------------------------------------------------------
diff --git a/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java b/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
index 2f2ea46..8f170af 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
@@ -18,9 +18,9 @@
package org.apache.beam.examples;
+import com.google.common.base.Strings;
import java.util.Date;
import org.apache.beam.examples.WordCount.WordCountOptions;
-import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.FileChecksumMatcher;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -36,6 +36,8 @@ import org.junit.runners.JUnit4;
@RunWith(JUnit4.class)
public class WordCountIT {
+ private static final String DEFAULT_OUTPUT_CHECKSUM = "8ae94f799f97cfd1cb5e8125951b32dfb52e1f12";
+
/**
* Options for the WordCount Integration Test.
*
@@ -43,9 +45,8 @@ public class WordCountIT {
* with customized input.
*/
public interface WordCountITOptions extends TestPipelineOptions, WordCountOptions {
- @Default.String("c04722202dee29c442b55ead54c6000693e85e77")
- String getOutputChecksum();
- void setOutputChecksum(String value);
+ String getChecksum();
+ void setChecksum(String value);
}
@Test
@@ -58,11 +59,13 @@ public class WordCountIT {
String.format("WordCountIT-%tF-%<tH-%<tM-%<tS-%<tL", new Date()),
"output",
"results"));
- options.setOnSuccessMatcher(
- new FileChecksumMatcher(options.getOutputChecksum(), options.getOutput() + "*"));
- String e2eTestInputPath = "gs://apache-beam-samples/apache/LICENSE";
- options.setInputFile(e2eTestInputPath);
+ String outputChecksum =
+ Strings.isNullOrEmpty(options.getChecksum())
+ ? DEFAULT_OUTPUT_CHECKSUM
+ : options.getChecksum();
+ options.setOnSuccessMatcher(
+ new FileChecksumMatcher(outputChecksum, options.getOutput() + "*"));
WordCount.main(TestPipeline.convertToArgs(options));
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1eccd29b/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
----------------------------------------------------------------------
diff --git a/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java b/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
index 27a5a8f..209c713 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
@@ -18,8 +18,8 @@
package org.apache.beam.examples.cookbook;
+import com.google.common.base.Strings;
import org.apache.beam.sdk.options.BigQueryOptions;
-import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.BigqueryMatcher;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -34,12 +34,13 @@ import org.junit.runners.JUnit4;
@RunWith(JUnit4.class)
public class BigQueryTornadoesIT {
+ private static final String DEFAULT_OUTPUT_CHECKSUM = "1ab4c7ec460b94bbb3c3885b178bf0e6bed56e1f";
+
/**
* Options for the BigQueryTornadoes Integration Test.
*/
public interface BigQueryTornadoesITOptions
extends TestPipelineOptions, BigQueryTornadoes.Options, BigQueryOptions {
- @Default.String("1ab4c7ec460b94bbb3c3885b178bf0e6bed56e1f")
String getChecksum();
void setChecksum(String value);
}
@@ -54,9 +55,13 @@ public class BigQueryTornadoesIT {
String query =
String.format("SELECT month, tornado_count FROM [%s]", options.getOutput());
+ String outputChecksum =
+ Strings.isNullOrEmpty(options.getChecksum())
+ ? DEFAULT_OUTPUT_CHECKSUM
+ : options.getChecksum();
options.setOnSuccessMatcher(
new BigqueryMatcher(
- options.getAppName(), options.getProject(), query, options.getChecksum()));
+ options.getAppName(), options.getProject(), query, outputChecksum));
BigQueryTornadoes.main(TestPipeline.convertToArgs(options));
}
[18/50] incubator-beam git commit: Move the separate
package-info.java compile to java7 profile since it's not needed with java8
This fixes compiling in Eclipse (assuming Neon which require java8) Slightly
speeds up Java8 compile (one invoke of compiler
Posted by th...@apache.org.
Move the separate package-info.java compile to java7 profile since it's not needed with java8
This fixes compiling in Eclipse (assuming Neon which require java8)
Slightly speeds up Java8 compile (one invoke of compiler plugin)
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3cef71e0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3cef71e0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3cef71e0
Branch: refs/heads/apex-runner
Commit: 3cef71e0653c24c44aef3515ce80c70eceac37c6
Parents: eac68cb
Author: Daniel Kulp <dk...@apache.org>
Authored: Wed Nov 2 16:14:00 2016 -0400
Committer: Kenneth Knowles <kl...@google.com>
Committed: Thu Nov 3 15:10:56 2016 -0700
----------------------------------------------------------------------
pom.xml | 73 +++++++++++++++++++++++++++++++++++++++---------------------
1 file changed, 48 insertions(+), 25 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3cef71e0/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index cb1e4d6..ea7d4ae 100644
--- a/pom.xml
+++ b/pom.xml
@@ -133,6 +133,8 @@
<woodstox.version>4.4.1</woodstox.version>
<compiler.error.flag>-Werror</compiler.error.flag>
+ <compiler.default.pkginfo.flag>-Xpkginfo:always</compiler.default.pkginfo.flag>
+ <compiler.default.exclude>nothing</compiler.default.exclude>
</properties>
<packaging>pom</packaging>
@@ -224,6 +226,48 @@
<beam.javadoc_opts>-Xdoclint:-missing</beam.javadoc_opts>
</properties>
</profile>
+ <profile>
+ <id>java7-packageinfo</id>
+ <activation>
+ <jdk>1.7</jdk>
+ </activation>
+ <properties>
+ <!--
+ Exclude package-info.java from main compilation to work around
+ https://jira.codehaus.org/browse/MCOMPILER-205
+ -->
+ <compiler.default.pkginfo.flag>-Xpkginfo:legacy</compiler.default.pkginfo.flag>
+ <compiler.default.exclude>**/package-info.java</compiler.default.exclude>
+ </properties>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <executions>
+ <!--
+ Compile just package-info.java to avoid
+ https://bugs.openjdk.java.net/browse/JDK-8022161
+ -->
+ <execution>
+ <id>compile-package-info</id>
+ <goals>
+ <goal>compile</goal>
+ </goals>
+ <phase>compile</phase>
+ <configuration>
+ <compilerArgs>
+ <arg>-Xpkginfo:always</arg>
+ </compilerArgs>
+ <includes>
+ <include>**/package-info.java</include>
+ </includes>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
<profile>
<id>eclipse-jdt</id>
@@ -832,10 +876,6 @@
</configuration>
<executions>
- <!--
- Exclude package-info.java from main compilation to work around
- https://jira.codehaus.org/browse/MCOMPILER-205
- -->
<execution>
<id>default-compile</id>
<goals>
@@ -843,29 +883,12 @@
</goals>
<phase>compile</phase>
<configuration>
- <excludes>
- <exclude>**/package-info.java</exclude>
- </excludes>
- </configuration>
- </execution>
-
- <!--
- Compile just package-info.java to avoid
- https://bugs.openjdk.java.net/browse/JDK-8022161
- -->
- <execution>
- <id>compile-package-info</id>
- <goals>
- <goal>compile</goal>
- </goals>
- <phase>compile</phase>
- <configuration>
<compilerArgs>
- <arg>-Xpkginfo:always</arg>
+ <arg>${compiler.default.pkginfo.flag}</arg>
</compilerArgs>
- <includes>
- <include>**/package-info.java</include>
- </includes>
+ <excludes>
+ <exclude>${compiler.default.exclude}</exclude>
+ </excludes>
</configuration>
</execution>
</executions>
[21/50] incubator-beam git commit: This closes #1251
Posted by th...@apache.org.
This closes #1251
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b93ac70c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b93ac70c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b93ac70c
Branch: refs/heads/apex-runner
Commit: b93ac70c9dbff006b51fa75d47496ebd554b6976
Parents: 2a1fdee 6b36fea
Author: Thomas Groh <tg...@google.com>
Authored: Thu Nov 3 15:15:59 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Thu Nov 3 15:15:59 2016 -0700
----------------------------------------------------------------------
.../runners/direct/AggregatorContainer.java | 20 +-
.../GroupAlsoByWindowEvaluatorFactory.java | 248 +++++++++++++++----
2 files changed, 218 insertions(+), 50 deletions(-)
----------------------------------------------------------------------
[19/50] incubator-beam git commit: This closes #1265
Posted by th...@apache.org.
This closes #1265
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2a1fdee5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2a1fdee5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2a1fdee5
Branch: refs/heads/apex-runner
Commit: 2a1fdee5c617848e90896db5cafe685331132fef
Parents: eac68cb 3cef71e
Author: Kenneth Knowles <kl...@google.com>
Authored: Thu Nov 3 15:10:57 2016 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Thu Nov 3 15:10:57 2016 -0700
----------------------------------------------------------------------
pom.xml | 73 +++++++++++++++++++++++++++++++++++++++---------------------
1 file changed, 48 insertions(+), 25 deletions(-)
----------------------------------------------------------------------
[17/50] incubator-beam git commit: Move watermark updates to the end
of handleResult
Posted by th...@apache.org.
Move watermark updates to the end of handleResult
This ensures that any state modifications are visible before watermarks
advancement permits additional progress.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0cb3832e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0cb3832e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0cb3832e
Branch: refs/heads/apex-runner
Commit: 0cb3832eb21ffeebc33b433d57cb814957788037
Parents: 55d9519
Author: Thomas Groh <tg...@google.com>
Authored: Fri Oct 21 10:13:06 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Thu Nov 3 13:45:34 2016 -0700
----------------------------------------------------------------------
.../apache/beam/runners/direct/EvaluationContext.java | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0cb3832e/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
index e5a30d4..965e77d 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
@@ -179,11 +179,6 @@ class EvaluationContext {
: completedBundle.withElements((Iterable) result.getUnprocessedElements()),
committedBundles,
outputTypes);
- watermarkManager.updateWatermarks(
- completedBundle,
- result.getTimerUpdate().withCompletedTimers(completedTimers),
- committedResult,
- result.getWatermarkHold());
// Commit aggregator changes
if (result.getAggregatorChanges() != null) {
result.getAggregatorChanges().commit();
@@ -201,6 +196,13 @@ class EvaluationContext {
applicationStateInternals.remove(stepAndKey);
}
}
+ // Watermarks are updated last to ensure visibility of any global state before progress is
+ // permitted
+ watermarkManager.updateWatermarks(
+ completedBundle,
+ result.getTimerUpdate().withCompletedTimers(completedTimers),
+ committedResult,
+ result.getWatermarkHold());
return committedResult;
}
[49/50] incubator-beam git commit: Merge branch 'master' into
apex-runner
Posted by th...@apache.org.
Merge branch 'master' into apex-runner
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5ccd5fb3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5ccd5fb3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5ccd5fb3
Branch: refs/heads/apex-runner
Commit: 5ccd5fb3c62ebdf1310af4af17cab2270b4a0d96
Parents: 15287b8 1102455
Author: Thomas Weise <th...@apache.org>
Authored: Mon Nov 7 20:12:36 2016 +0100
Committer: Thomas Weise <th...@apache.org>
Committed: Mon Nov 7 20:12:36 2016 +0100
----------------------------------------------------------------------
.travis.yml | 6 +-
examples/java/pom.xml | 79 --
.../beam/examples/common/ExampleUtils.java | 3 +-
.../apache/beam/examples/complete/TfIdf.java | 6 +-
.../beam/examples/cookbook/DeDupExample.java | 8 +-
.../org/apache/beam/examples/cookbook/README.md | 2 +-
.../beam/examples/WindowedWordCountIT.java | 11 +-
.../org/apache/beam/examples/WordCountIT.java | 19 +-
.../beam/examples/complete/TfIdfTest.java | 4 +-
.../examples/cookbook/BigQueryTornadoesIT.java | 11 +-
.../examples/cookbook/DeDupExampleTest.java | 82 --
.../examples/cookbook/DistinctExampleTest.java | 82 ++
examples/java8/pom.xml | 19 -
.../beam/examples/complete/game/GameStats.java | 3 +-
.../examples/complete/game/LeaderBoard.java | 2 +-
.../beam/examples/complete/game/UserScore.java | 2 +-
.../examples/MinimalWordCountJava8Test.java | 2 +-
examples/pom.xml | 13 +-
pom.xml | 107 ++-
runners/core-java/pom.xml | 65 --
.../core/GroupAlsoByWindowViaWindowSetDoFn.java | 3 +-
.../GroupAlsoByWindowsViaOutputBufferDoFn.java | 3 +-
.../core/GroupByKeyViaGroupByKeyOnly.java | 13 +-
.../core/PushbackSideInputDoFnRunner.java | 28 +-
.../beam/runners/core/ReduceFnTester.java | 6 +-
.../UnboundedReadFromBoundedSourceTest.java | 4 +-
runners/direct-java/pom.xml | 102 ---
.../runners/direct/AggregatorContainer.java | 20 +-
...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 16 +-
.../beam/runners/direct/DirectGroupByKey.java | 42 +-
.../direct/DirectGroupByKeyOverrideFactory.java | 25 +-
.../runners/direct/DirectTimerInternals.java | 13 +
.../beam/runners/direct/EvaluationContext.java | 18 +-
.../direct/ExecutorServiceParallelExecutor.java | 41 +-
.../GroupAlsoByWindowEvaluatorFactory.java | 249 ++++--
.../direct/GroupByKeyOnlyEvaluatorFactory.java | 31 +-
.../direct/PTransformOverrideFactory.java | 8 +-
.../beam/runners/direct/ParDoEvaluator.java | 28 +-
.../runners/direct/ParDoEvaluatorFactory.java | 126 ++++
.../direct/ParDoMultiEvaluatorFactory.java | 107 ---
.../direct/ParDoMultiEvaluatorHooks.java | 55 ++
.../runners/direct/ParDoOverrideFactory.java | 14 +-
.../direct/ParDoSingleEvaluatorFactory.java | 110 ---
.../direct/ParDoSingleEvaluatorHooks.java | 58 ++
.../direct/TestStreamEvaluatorFactory.java | 14 +-
.../direct/TransformEvaluatorRegistry.java | 10 +-
.../direct/UncommittedBundleOutputManager.java | 50 --
.../runners/direct/ViewEvaluatorFactory.java | 19 +-
.../direct/WatermarkCallbackExecutor.java | 3 +-
.../beam/runners/direct/WatermarkManager.java | 79 +-
.../direct/WriteWithShardingFactory.java | 15 +-
.../runners/direct/EvaluationContextTest.java | 23 +-
.../direct/GroupByKeyEvaluatorFactoryTest.java | 29 +-
.../GroupByKeyOnlyEvaluatorFactoryTest.java | 31 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 16 +-
.../direct/ParDoMultiEvaluatorFactoryTest.java | 439 -----------
.../direct/ParDoMultiEvaluatorHooksTest.java | 439 +++++++++++
.../direct/ParDoSingleEvaluatorFactoryTest.java | 335 ---------
.../direct/ParDoSingleEvaluatorHooksTest.java | 335 +++++++++
.../runners/direct/WatermarkManagerTest.java | 102 +--
.../direct/WriteWithShardingFactoryTest.java | 26 +-
runners/flink/examples/pom.xml | 17 -
.../beam/runners/flink/examples/TFIDF.java | 6 +-
.../flink/examples/streaming/AutoComplete.java | 8 +-
.../flink/examples/streaming/JoinExamples.java | 12 +-
.../examples/streaming/KafkaIOExamples.java | 2 +-
.../examples/streaming/WindowedWordCount.java | 8 +-
runners/flink/runner/pom.xml | 17 -
.../wrappers/streaming/DoFnOperator.java | 10 +-
.../wrappers/streaming/WindowDoFnOperator.java | 13 +
runners/google-cloud-dataflow-java/pom.xml | 114 ---
.../dataflow/BlockingDataflowRunner.java | 170 -----
.../dataflow/DataflowJobCancelledException.java | 39 -
.../dataflow/DataflowJobExecutionException.java | 35 -
.../dataflow/DataflowJobUpdatedException.java | 51 --
.../runners/dataflow/DataflowPipelineJob.java | 44 +-
.../dataflow/DataflowPipelineRegistrar.java | 11 +-
.../dataflow/DataflowPipelineTranslator.java | 8 +-
.../beam/runners/dataflow/DataflowRunner.java | 49 +-
.../runners/dataflow/internal/IsmFormat.java | 5 +-
.../dataflow/internal/ReadTranslator.java | 12 +-
.../BlockingDataflowPipelineOptions.java | 28 -
.../options/DataflowPipelineOptions.java | 8 +
.../testing/TestDataflowPipelineOptions.java | 6 +-
.../dataflow/testing/TestDataflowRunner.java | 7 +-
.../dataflow/util/DataflowTemplateJob.java | 70 ++
.../beam/runners/dataflow/util/DoFnInfo.java | 18 +-
.../runners/dataflow/util/MonitoringUtil.java | 1 -
.../runners/dataflow/util/RandomAccessData.java | 2 +-
.../dataflow/BlockingDataflowRunnerTest.java | 300 --------
.../dataflow/DataflowPipelineJobTest.java | 30 +-
.../dataflow/DataflowPipelineRegistrarTest.java | 7 +-
.../DataflowPipelineTranslatorTest.java | 28 +
.../runners/dataflow/DataflowRunnerTest.java | 45 ++
runners/pom.xml | 11 +
runners/spark/pom.xml | 12 -
.../runners/spark/SparkPipelineOptions.java | 3 +-
.../apache/beam/runners/spark/io/SourceRDD.java | 4 +-
.../translation/GroupCombineFunctions.java | 5 +-
.../ReifyTimestampsAndWindowsFunction.java | 47 ++
.../spark/translation/SparkProcessContext.java | 46 +-
.../SparkRunnerStreamingContextFactory.java | 23 +-
.../streaming/EmptyStreamAssertionTest.java | 3 +-
.../streaming/FlattenStreamingTest.java | 6 +-
.../streaming/KafkaStreamingTest.java | 10 +-
.../ResumeFromCheckpointStreamingTest.java | 3 +-
.../streaming/SimpleStreamingWordCountTest.java | 3 +-
.../utils/TestOptionsForStreaming.java | 12 +-
sdks/java/build-tools/pom.xml | 4 +-
.../src/main/resources/beam/checkstyle.xml | 9 +-
sdks/java/core/pom.xml | 133 +---
.../java/org/apache/beam/sdk/io/AvroIO.java | 4 +-
.../sdk/io/BoundedReadFromUnboundedSource.java | 4 +-
.../org/apache/beam/sdk/io/BoundedSource.java | 11 +-
.../apache/beam/sdk/io/CompressedSource.java | 4 +-
.../org/apache/beam/sdk/io/FileBasedSink.java | 43 +-
.../org/apache/beam/sdk/io/FileBasedSource.java | 59 +-
.../beam/sdk/io/PubsubUnboundedSource.java | 123 ++-
.../main/java/org/apache/beam/sdk/io/Sink.java | 4 +-
.../java/org/apache/beam/sdk/io/TextIO.java | 43 +-
.../java/org/apache/beam/sdk/io/XmlSource.java | 2 +-
.../apache/beam/sdk/io/range/ByteKeyRange.java | 2 +-
.../sdk/options/PipelineOptionsFactory.java | 248 ++++--
.../sdk/options/ProxyInvocationHandler.java | 5 +-
.../apache/beam/sdk/options/ValueProvider.java | 38 +-
.../apache/beam/sdk/transforms/Aggregator.java | 13 +-
.../apache/beam/sdk/transforms/CombineFns.java | 4 +-
.../apache/beam/sdk/transforms/Distinct.java | 158 ++++
.../org/apache/beam/sdk/transforms/DoFn.java | 22 +-
.../org/apache/beam/sdk/transforms/Flatten.java | 3 +-
.../apache/beam/sdk/transforms/GroupByKey.java | 4 +-
.../org/apache/beam/sdk/transforms/ParDo.java | 2 +-
.../org/apache/beam/sdk/transforms/Regex.java | 454 +++++++++++
.../beam/sdk/transforms/RemoveDuplicates.java | 159 ----
.../org/apache/beam/sdk/transforms/View.java | 2 +-
.../sdk/transforms/display/DisplayData.java | 5 +-
.../beam/sdk/transforms/join/CoGroupByKey.java | 6 +-
.../sdk/transforms/reflect/DoFnInvokers.java | 7 +-
.../sdk/transforms/reflect/DoFnSignature.java | 46 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 752 ++++++++++++-------
.../splittabledofn/RestrictionTracker.java | 2 +-
.../transforms/splittabledofn/package-info.java | 4 +-
.../beam/sdk/transforms/windowing/Window.java | 130 +---
.../beam/sdk/util/BaseExecutionContext.java | 2 +-
.../BufferedElementCountingOutputStream.java | 4 +-
.../apache/beam/sdk/util/ExecutableTrigger.java | 131 ----
.../apache/beam/sdk/util/ExecutionContext.java | 8 +-
.../sdk/util/ExposedByteArrayInputStream.java | 5 +-
.../sdk/util/ExposedByteArrayOutputStream.java | 2 -
.../beam/sdk/util/FileIOChannelFactory.java | 8 +-
.../apache/beam/sdk/util/GatherAllPanes.java | 18 +-
.../beam/sdk/util/GcsIOChannelFactory.java | 8 +-
.../java/org/apache/beam/sdk/util/GcsUtil.java | 2 -
.../apache/beam/sdk/util/IOChannelFactory.java | 4 +
.../apache/beam/sdk/util/MovingFunction.java | 4 +-
.../org/apache/beam/sdk/util/PubsubClient.java | 32 +-
.../apache/beam/sdk/util/PubsubTestClient.java | 32 +
.../sdk/util/ReifyTimestampAndWindowsDoFn.java | 41 -
.../sdk/util/ReifyTimestampsAndWindows.java | 63 --
.../org/apache/beam/sdk/util/StringUtils.java | 8 +-
.../apache/beam/sdk/util/TimerInternals.java | 35 +-
.../apache/beam/sdk/util/WindowingStrategy.java | 10 +-
.../beam/sdk/util/common/ReflectHelpers.java | 17 +
.../org/apache/beam/sdk/util/gcsfs/GcsPath.java | 13 +-
.../sdk/util/state/InMemoryTimerInternals.java | 12 +
.../beam/sdk/util/state/StateNamespace.java | 16 +-
.../java/org/apache/beam/sdk/values/PDone.java | 2 +-
.../apache/beam/sdk/values/TypeDescriptor.java | 48 +-
.../apache/beam/sdk/values/TypeDescriptors.java | 1 -
.../apache/beam/sdk/values/TypeParameter.java | 62 ++
.../apache/beam/sdk/io/CountingInputTest.java | 6 +-
.../apache/beam/sdk/io/CountingSourceTest.java | 8 +-
.../apache/beam/sdk/io/FileBasedSinkTest.java | 23 +-
.../beam/sdk/io/PubsubUnboundedSourceTest.java | 79 ++
.../org/apache/beam/sdk/io/XmlSinkTest.java | 16 +-
.../sdk/options/PipelineOptionsFactoryTest.java | 239 ++++++
.../sdk/options/ProxyInvocationHandlerTest.java | 29 +-
.../beam/sdk/options/ValueProviderTest.java | 33 +
.../apache/beam/sdk/testing/PAssertTest.java | 27 +
.../sdk/transforms/ApproximateUniqueTest.java | 2 +-
.../beam/sdk/transforms/DistinctTest.java | 130 ++++
.../apache/beam/sdk/transforms/RegexTest.java | 248 ++++++
.../sdk/transforms/RemoveDuplicatesTest.java | 130 ----
.../DoFnSignaturesProcessElementTest.java | 18 +-
.../DoFnSignaturesSplittableDoFnTest.java | 19 +-
.../transforms/reflect/DoFnSignaturesTest.java | 42 +-
.../reflect/DoFnSignaturesTestUtils.java | 13 +-
.../sdk/transforms/windowing/WindowTest.java | 6 +-
.../beam/sdk/util/ExecutableTriggerTest.java | 109 ---
.../sdk/util/common/ReflectHelpersTest.java | 25 +
.../apache/beam/sdk/util/gcsfs/GcsPathTest.java | 25 +
.../beam/sdk/values/TypeDescriptorTest.java | 12 +
sdks/java/extensions/join-library/pom.xml | 8 -
sdks/java/extensions/pom.xml | 1 +
sdks/java/extensions/sorter/README.md | 42 ++
sdks/java/extensions/sorter/pom.xml | 159 ++++
.../sorter/BufferedExternalSorter.java | 125 +++
.../sdk/extensions/sorter/ExternalSorter.java | 225 ++++++
.../sdk/extensions/sorter/InMemorySorter.java | 166 ++++
.../beam/sdk/extensions/sorter/SortValues.java | 213 ++++++
.../beam/sdk/extensions/sorter/Sorter.java | 47 ++
.../sdk/extensions/sorter/package-info.java | 23 +
.../sorter/BufferedExternalSorterTest.java | 177 +++++
.../extensions/sorter/ExternalSorterTest.java | 87 +++
.../extensions/sorter/InMemorySorterTest.java | 144 ++++
.../sdk/extensions/sorter/SortValuesTest.java | 128 ++++
.../sdk/extensions/sorter/SorterTestUtils.java | 129 ++++
sdks/java/io/google-cloud-platform/pom.xml | 8 -
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 2 +-
.../apache/beam/sdk/io/gcp/ApiSurfaceTest.java | 132 ++++
sdks/java/io/hdfs/pom.xml | 8 -
sdks/java/io/jdbc/pom.xml | 14 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIO.java | 39 +
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 33 +
sdks/java/io/jms/pom.xml | 8 -
sdks/java/io/kafka/pom.xml | 15 -
.../apache/beam/sdk/io/kafka/KafkaIOTest.java | 6 +-
sdks/java/io/kinesis/pom.xml | 15 -
.../apache/beam/sdk/io/kinesis/KinesisIO.java | 35 +-
sdks/java/io/mongodb/pom.xml | 12 -
sdks/java/java8tests/pom.xml | 11 -
.../beam/sdk/transforms/DistinctJava8Test.java | 104 +++
.../transforms/RemoveDuplicatesJava8Test.java | 104 ---
.../main/resources/archetype-resources/pom.xml | 32 -
sdks/java/microbenchmarks/pom.xml | 10 -
sdks/java/pom.xml | 9 +-
sdks/pom.xml | 16 +-
227 files changed, 7001 insertions(+), 4602 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5ccd5fb3/examples/java/pom.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5ccd5fb3/runners/pom.xml
----------------------------------------------------------------------
[31/50] incubator-beam git commit: pom: move maven-archetypes to its
proper spot in dep hierarchy
Posted by th...@apache.org.
pom: move maven-archetypes to its proper spot in dep hierarchy
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f19a25da
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f19a25da
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f19a25da
Branch: refs/heads/apex-runner
Commit: f19a25da225b65ecdf43342ec48389a8348e68ce
Parents: f06deac
Author: Dan Halperin <dh...@google.com>
Authored: Thu Nov 3 17:30:58 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Fri Nov 4 10:56:39 2016 -0700
----------------------------------------------------------------------
pom.xml | 4 ----
sdks/java/pom.xml | 5 +----
2 files changed, 1 insertion(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f19a25da/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 28845a5..14c38fd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -145,10 +145,6 @@
<module>sdks/java/build-tools</module>
<module>sdks</module>
<module>runners</module>
- <!-- sdks/java/maven-archetypes has several dependencies on the DataflowPipelineRunner.
- Until these are refactored out or a released artifact exists, we need to modify the
- build order. -->
- <module>sdks/java/maven-archetypes</module>
<module>examples</module>
</modules>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f19a25da/sdks/java/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/pom.xml b/sdks/java/pom.xml
index 2866cb3..fb07d25 100644
--- a/sdks/java/pom.xml
+++ b/sdks/java/pom.xml
@@ -38,10 +38,7 @@
<module>build-tools</module> -->
<module>core</module>
<module>io</module>
- <!-- sdks/java/maven-archtypes has several dependencies on the
- DataflowRunner. Until these are refactored out or
- a released artifact exists, we need to modify the build order.
- <module>maven-archetypes</module> -->
+ <module>maven-archetypes</module>
<module>extensions</module>
<module>microbenchmarks</module>
</modules>
[07/50] incubator-beam git commit: This closes #1259
Posted by th...@apache.org.
This closes #1259
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d75d8b2b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d75d8b2b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d75d8b2b
Branch: refs/heads/apex-runner
Commit: d75d8b2bb7370f7b0527da73937f079b5b8ee09c
Parents: 4dec38c 8c2b99e
Author: Davor Bonaci <da...@google.com>
Authored: Wed Nov 2 16:51:08 2016 -0700
Committer: Davor Bonaci <da...@google.com>
Committed: Wed Nov 2 16:51:08 2016 -0700
----------------------------------------------------------------------
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
[02/50] incubator-beam git commit: This closes #1225
Posted by th...@apache.org.
This closes #1225
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/52246933
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/52246933
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/52246933
Branch: refs/heads/apex-runner
Commit: 522469330dff8a65ff2e3d9b411c00a42c031864
Parents: 529f266 e77d881
Author: Thomas Groh <tg...@google.com>
Authored: Wed Nov 2 10:18:59 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Wed Nov 2 10:18:59 2016 -0700
----------------------------------------------------------------------
.../core/GroupByKeyViaGroupByKeyOnly.java | 13 ++--
...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 2 -
.../beam/runners/direct/DirectGroupByKey.java | 42 ++++++-------
.../GroupAlsoByWindowEvaluatorFactory.java | 4 +-
.../direct/GroupByKeyOnlyEvaluatorFactory.java | 31 +++++-----
.../beam/runners/direct/ParDoEvaluator.java | 5 +-
.../direct/ParDoMultiEvaluatorHooks.java | 1 +
.../direct/ParDoSingleEvaluatorHooks.java | 1 +
.../direct/GroupByKeyEvaluatorFactoryTest.java | 29 ++++-----
.../GroupByKeyOnlyEvaluatorFactoryTest.java | 31 +++++-----
.../beam/runners/direct/ParDoEvaluatorTest.java | 5 +-
.../apache/beam/sdk/util/GatherAllPanes.java | 18 +++++-
.../sdk/util/ReifyTimestampAndWindowsDoFn.java | 41 -------------
.../sdk/util/ReifyTimestampsAndWindows.java | 63 --------------------
14 files changed, 94 insertions(+), 192 deletions(-)
----------------------------------------------------------------------
[42/50] incubator-beam git commit: move source jar creation into
release profile
Posted by th...@apache.org.
move source jar creation into release profile
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/68f26386
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/68f26386
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/68f26386
Branch: refs/heads/apex-runner
Commit: 68f26386e6ba7c5a883ba34accccff4618234811
Parents: 5864a38
Author: Dan Halperin <dh...@google.com>
Authored: Sat Nov 5 01:45:57 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Mon Nov 7 08:45:50 2016 -0800
----------------------------------------------------------------------
examples/java/pom.xml | 6 ------
examples/java8/pom.xml | 6 ------
pom.xml | 5 +++++
runners/core-java/pom.xml | 6 ------
runners/direct-java/pom.xml | 6 ------
runners/flink/examples/pom.xml | 4 ----
runners/flink/runner/pom.xml | 4 ----
runners/google-cloud-dataflow-java/pom.xml | 6 ------
runners/spark/pom.xml | 4 ----
sdks/java/core/pom.xml | 6 ------
sdks/java/extensions/join-library/pom.xml | 4 ----
sdks/java/extensions/sorter/pom.xml | 4 ----
sdks/java/io/google-cloud-platform/pom.xml | 4 ----
sdks/java/io/hdfs/pom.xml | 4 ----
sdks/java/io/jdbc/pom.xml | 4 ----
sdks/java/io/jms/pom.xml | 4 ----
sdks/java/io/kafka/pom.xml | 4 ----
sdks/java/io/kinesis/pom.xml | 4 ----
sdks/java/io/mongodb/pom.xml | 4 ----
sdks/java/java8tests/pom.xml | 6 ------
sdks/java/microbenchmarks/pom.xml | 5 -----
21 files changed, 5 insertions(+), 95 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/examples/java/pom.xml
----------------------------------------------------------------------
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 857373a..5d69bfe 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -277,12 +277,6 @@
</configuration>
</plugin>
- <!-- Source plugin for generating source and test-source JARs. -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/examples/java8/pom.xml
----------------------------------------------------------------------
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 0a5dcb0..72f24ca 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -140,12 +140,6 @@
</configuration>
</plugin>
- <!-- Source plugin for generating source and test-source JARs. -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 77e87ba..c1bd5c8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -220,6 +220,11 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
</plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-source-plugin</artifactId>
+ </plugin>
</plugins>
</build>
</profile>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/runners/core-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index 3d8a84b..40ebf58 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -57,12 +57,6 @@
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <!-- Source plugin for generating source and test-source JARs. -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/runners/direct-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index 64e9b5a..8983b1c 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -51,12 +51,6 @@
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <!-- Source plugin for generating source and test-source JARs. -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/runners/flink/examples/pom.xml
----------------------------------------------------------------------
diff --git a/runners/flink/examples/pom.xml b/runners/flink/examples/pom.xml
index 051ec2d..dccb5cb 100644
--- a/runners/flink/examples/pom.xml
+++ b/runners/flink/examples/pom.xml
@@ -90,10 +90,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/runners/flink/runner/pom.xml
----------------------------------------------------------------------
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 9729316..adcb3de 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -223,10 +223,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/runners/google-cloud-dataflow-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/google-cloud-dataflow-java/pom.xml b/runners/google-cloud-dataflow-java/pom.xml
index 0a2c9f6..a5ffe0e 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -77,12 +77,6 @@
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <!-- Source plugin for generating source and test-source JARs. -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/runners/spark/pom.xml
----------------------------------------------------------------------
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index a3ca296..5e94848 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -411,10 +411,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
</plugin>
</plugins>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/core/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index 7d4d9ab..7906afb 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -99,12 +99,6 @@
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <!-- Source plugin for generating source and test-source JARs. -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/extensions/join-library/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/extensions/join-library/pom.xml b/sdks/java/extensions/join-library/pom.xml
index 16326b0..554792a 100644
--- a/sdks/java/extensions/join-library/pom.xml
+++ b/sdks/java/extensions/join-library/pom.xml
@@ -46,10 +46,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
</plugins>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/extensions/sorter/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/extensions/sorter/pom.xml b/sdks/java/extensions/sorter/pom.xml
index 1a994de..8d1e89d 100644
--- a/sdks/java/extensions/sorter/pom.xml
+++ b/sdks/java/extensions/sorter/pom.xml
@@ -50,10 +50,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/io/google-cloud-platform/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/google-cloud-platform/pom.xml b/sdks/java/io/google-cloud-platform/pom.xml
index fa51072..9f7af52 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -52,10 +52,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/io/hdfs/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/hdfs/pom.xml b/sdks/java/io/hdfs/pom.xml
index e79996f..a551455 100644
--- a/sdks/java/io/hdfs/pom.xml
+++ b/sdks/java/io/hdfs/pom.xml
@@ -47,10 +47,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
</plugins>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/io/jdbc/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/pom.xml b/sdks/java/io/jdbc/pom.xml
index 2eb24bd..d48ff66 100644
--- a/sdks/java/io/jdbc/pom.xml
+++ b/sdks/java/io/jdbc/pom.xml
@@ -38,10 +38,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
</plugin>
<plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/io/jms/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/jms/pom.xml b/sdks/java/io/jms/pom.xml
index 0b46154..7063ce9 100644
--- a/sdks/java/io/jms/pom.xml
+++ b/sdks/java/io/jms/pom.xml
@@ -46,10 +46,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
</plugins>
</build>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/io/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/kafka/pom.xml b/sdks/java/io/kafka/pom.xml
index 2004952..1ad31aa 100644
--- a/sdks/java/io/kafka/pom.xml
+++ b/sdks/java/io/kafka/pom.xml
@@ -37,10 +37,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/io/kinesis/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/kinesis/pom.xml b/sdks/java/io/kinesis/pom.xml
index 439995d..f874c21 100644
--- a/sdks/java/io/kinesis/pom.xml
+++ b/sdks/java/io/kinesis/pom.xml
@@ -37,10 +37,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/io/mongodb/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/mongodb/pom.xml b/sdks/java/io/mongodb/pom.xml
index d0ae6ac..4b100a9 100644
--- a/sdks/java/io/mongodb/pom.xml
+++ b/sdks/java/io/mongodb/pom.xml
@@ -38,10 +38,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
</plugin>
<plugin>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/java8tests/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/java8tests/pom.xml b/sdks/java/java8tests/pom.xml
index 989fba2..d9dfdce 100644
--- a/sdks/java/java8tests/pom.xml
+++ b/sdks/java/java8tests/pom.xml
@@ -61,12 +61,6 @@
</configuration>
</plugin>
- <!-- Source plugin for generating source and test-source JARs. -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
-
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/68f26386/sdks/java/microbenchmarks/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/microbenchmarks/pom.xml b/sdks/java/microbenchmarks/pom.xml
index 05f06aa..c95309c 100644
--- a/sdks/java/microbenchmarks/pom.xml
+++ b/sdks/java/microbenchmarks/pom.xml
@@ -40,11 +40,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
[08/50] incubator-beam git commit: Revert "[BEAM-808] Increase
"spark.port.maxRetries" to avoid BindException in ROS."
Posted by th...@apache.org.
Revert "[BEAM-808] Increase "spark.port.maxRetries" to avoid BindException in ROS."
This reverts commit ff5409f15a9f741f437c489b7de763cfa3c68278.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f233d858
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f233d858
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f233d858
Branch: refs/heads/apex-runner
Commit: f233d858810cacace21b05e525131f05b26e6380
Parents: d75d8b2
Author: Sela <an...@paypal.com>
Authored: Thu Nov 3 14:13:33 2016 +0200
Committer: Sela <an...@paypal.com>
Committed: Thu Nov 3 14:13:33 2016 +0200
----------------------------------------------------------------------
runners/spark/pom.xml | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f233d858/runners/spark/pom.xml
----------------------------------------------------------------------
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index e83aedf..71a3ac2 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -87,7 +87,6 @@
</beamTestPipelineOptions>
<beam.spark.test.reuseSparkContext>true</beam.spark.test.reuseSparkContext>
<spark.ui.enabled>false</spark.ui.enabled>
- <spark.port.maxRetries>64</spark.port.maxRetries>
</systemPropertyVariables>
</configuration>
</execution>
[34/50] incubator-beam git commit: Flatten FiredTimers and
ExtractFiredTimers
Posted by th...@apache.org.
Flatten FiredTimers and ExtractFiredTimers
Pass a single collection of fired timers, and have those objects contain
the associated transform and key that they fired for. Timers already
contain the domain they are in.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5dca2674
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5dca2674
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5dca2674
Branch: refs/heads/apex-runner
Commit: 5dca2674a8d145c6e619005c2282c6064cd7aab7
Parents: 6e1e57b
Author: Thomas Groh <tg...@google.com>
Authored: Thu Nov 3 14:10:37 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Fri Nov 4 13:05:21 2016 -0700
----------------------------------------------------------------------
.../beam/runners/direct/EvaluationContext.java | 6 +-
.../direct/ExecutorServiceParallelExecutor.java | 41 ++++----
.../beam/runners/direct/WatermarkManager.java | 79 ++++++++------
.../runners/direct/EvaluationContextTest.java | 23 ++---
.../runners/direct/WatermarkManagerTest.java | 102 ++++++-------------
5 files changed, 109 insertions(+), 142 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5dca2674/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
index 965e77d..b814def 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
@@ -391,11 +391,9 @@ class EvaluationContext {
* <p>This is a destructive operation. Timers will only appear in the result of this method once
* for each time they are set.
*/
- public Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> extractFiredTimers() {
+ public Collection<FiredTimers> extractFiredTimers() {
forceRefresh();
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> fired =
- watermarkManager.extractFiredTimers();
- return fired;
+ return watermarkManager.extractFiredTimers();
}
/**
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5dca2674/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index e32f671..d1ffea1 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -47,7 +47,6 @@ import org.apache.beam.sdk.transforms.AppliedPTransform;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.util.KeyedWorkItem;
import org.apache.beam.sdk.util.KeyedWorkItems;
-import org.apache.beam.sdk.util.TimeDomain;
import org.apache.beam.sdk.util.TimerInternals.TimerData;
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.util.WindowedValue;
@@ -440,29 +439,23 @@ final class ExecutorServiceParallelExecutor implements PipelineExecutor {
*/
private void fireTimers() throws Exception {
try {
- for (Map.Entry<
- AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> transformTimers :
- evaluationContext.extractFiredTimers().entrySet()) {
- AppliedPTransform<?, ?, ?> transform = transformTimers.getKey();
- for (Map.Entry<StructuralKey<?>, FiredTimers> keyTimers :
- transformTimers.getValue().entrySet()) {
- for (TimeDomain domain : TimeDomain.values()) {
- Collection<TimerData> delivery = keyTimers.getValue().getTimers(domain);
- if (delivery.isEmpty()) {
- continue;
- }
- KeyedWorkItem<?, Object> work =
- KeyedWorkItems.timersWorkItem(keyTimers.getKey().getKey(), delivery);
- @SuppressWarnings({"unchecked", "rawtypes"})
- CommittedBundle<?> bundle =
- evaluationContext
- .createKeyedBundle(keyTimers.getKey(), (PCollection) transform.getInput())
- .add(WindowedValue.valueInGlobalWindow(work))
- .commit(evaluationContext.now());
- scheduleConsumption(transform, bundle, new TimerIterableCompletionCallback(delivery));
- state.set(ExecutorState.ACTIVE);
- }
- }
+ for (FiredTimers transformTimers : evaluationContext.extractFiredTimers()) {
+ Collection<TimerData> delivery = transformTimers.getTimers();
+ KeyedWorkItem<?, Object> work =
+ KeyedWorkItems.timersWorkItem(transformTimers.getKey().getKey(), delivery);
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ CommittedBundle<?> bundle =
+ evaluationContext
+ .createKeyedBundle(
+ transformTimers.getKey(),
+ (PCollection) transformTimers.getTransform().getInput())
+ .add(WindowedValue.valueInGlobalWindow(work))
+ .commit(evaluationContext.now());
+ scheduleConsumption(
+ transformTimers.getTransform(),
+ bundle,
+ new TimerIterableCompletionCallback(delivery));
+ state.set(ExecutorState.ACTIVE);
}
} catch (Exception e) {
LOG.error("Internal Error while delivering timers", e);
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5dca2674/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
index 31b8091..f01c13c 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
@@ -740,14 +740,17 @@ public class WatermarkManager {
wms =
new TransformWatermarks(
- inputWatermark, outputWatermark, inputProcessingWatermark, outputProcessingWatermark);
+ transform,
+ inputWatermark,
+ outputWatermark,
+ inputProcessingWatermark,
+ outputProcessingWatermark);
transformToWatermarks.put(transform, wms);
}
return wms;
}
- private Collection<Watermark> getInputProcessingWatermarks(
- AppliedPTransform<?, ?, ?> transform) {
+ private Collection<Watermark> getInputProcessingWatermarks(AppliedPTransform<?, ?, ?> transform) {
ImmutableList.Builder<Watermark> inputWmsBuilder = ImmutableList.builder();
Collection<? extends PValue> inputs = transform.getInput().expand();
if (inputs.isEmpty()) {
@@ -924,15 +927,12 @@ public class WatermarkManager {
* Returns a map of each {@link PTransform} that has pending timers to those timers. All of the
* pending timers will be removed from this {@link WatermarkManager}.
*/
- public Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> extractFiredTimers() {
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> allTimers = new HashMap<>();
+ public Collection<FiredTimers> extractFiredTimers() {
+ Collection<FiredTimers> allTimers = new ArrayList<>();
for (Map.Entry<AppliedPTransform<?, ?, ?>, TransformWatermarks> watermarksEntry :
transformToWatermarks.entrySet()) {
- Map<StructuralKey<?>, FiredTimers> keyFiredTimers =
- watermarksEntry.getValue().extractFiredTimers();
- if (!keyFiredTimers.isEmpty()) {
- allTimers.put(watermarksEntry.getKey(), keyFiredTimers);
- }
+ Collection<FiredTimers> firedTimers = watermarksEntry.getValue().extractFiredTimers();
+ allTimers.addAll(firedTimers);
}
return allTimers;
}
@@ -1043,6 +1043,8 @@ public class WatermarkManager {
* A reference to the input and output watermarks of an {@link AppliedPTransform}.
*/
public class TransformWatermarks {
+ private final AppliedPTransform<?, ?, ?> transform;
+
private final AppliedPTransformInputWatermark inputWatermark;
private final AppliedPTransformOutputWatermark outputWatermark;
@@ -1053,10 +1055,12 @@ public class WatermarkManager {
private Instant latestSynchronizedOutputWm;
private TransformWatermarks(
+ AppliedPTransform<?, ?, ?> transform,
AppliedPTransformInputWatermark inputWatermark,
AppliedPTransformOutputWatermark outputWatermark,
SynchronizedProcessingTimeInputWatermark inputSynchProcessingWatermark,
SynchronizedProcessingTimeOutputWatermark outputSynchProcessingWatermark) {
+ this.transform = transform;
this.inputWatermark = inputWatermark;
this.outputWatermark = outputWatermark;
@@ -1128,7 +1132,7 @@ public class WatermarkManager {
synchronizedProcessingInputWatermark.addPending(bundle);
}
- private Map<StructuralKey<?>, FiredTimers> extractFiredTimers() {
+ private Collection<FiredTimers> extractFiredTimers() {
Map<StructuralKey<?>, List<TimerData>> eventTimeTimers =
inputWatermark.extractFiredEventTimeTimers();
Map<StructuralKey<?>, List<TimerData>> processingTimers;
@@ -1137,31 +1141,33 @@ public class WatermarkManager {
TimeDomain.PROCESSING_TIME, clock.now());
synchronizedTimers = synchronizedProcessingInputWatermark.extractFiredDomainTimers(
TimeDomain.SYNCHRONIZED_PROCESSING_TIME, getSynchronizedProcessingInputTime());
- Map<StructuralKey<?>, Map<TimeDomain, List<TimerData>>> groupedTimers = new HashMap<>();
- groupFiredTimers(groupedTimers, eventTimeTimers, processingTimers, synchronizedTimers);
- Map<StructuralKey<?>, FiredTimers> keyFiredTimers = new HashMap<>();
- for (Map.Entry<StructuralKey<?>, Map<TimeDomain, List<TimerData>>> firedTimers :
- groupedTimers.entrySet()) {
- keyFiredTimers.put(firedTimers.getKey(), new FiredTimers(firedTimers.getValue()));
+ Map<StructuralKey<?>, List<TimerData>> timersPerKey =
+ groupFiredTimers(eventTimeTimers, processingTimers, synchronizedTimers);
+ Collection<FiredTimers> keyFiredTimers = new ArrayList<>(timersPerKey.size());
+ for (Map.Entry<StructuralKey<?>, List<TimerData>> firedTimers :
+ timersPerKey.entrySet()) {
+ keyFiredTimers.add(
+ new FiredTimers(transform, firedTimers.getKey(), firedTimers.getValue()));
}
return keyFiredTimers;
}
@SafeVarargs
- private final void groupFiredTimers(
- Map<StructuralKey<?>, Map<TimeDomain, List<TimerData>>> groupedToMutate,
+ private final Map<StructuralKey<?>, List<TimerData>> groupFiredTimers(
Map<StructuralKey<?>, List<TimerData>>... timersToGroup) {
+ Map<StructuralKey<?>, List<TimerData>> groupedTimers = new HashMap<>();
for (Map<StructuralKey<?>, List<TimerData>> subGroup : timersToGroup) {
for (Map.Entry<StructuralKey<?>, List<TimerData>> newTimers : subGroup.entrySet()) {
- Map<TimeDomain, List<TimerData>> grouped = groupedToMutate.get(newTimers.getKey());
+ List<TimerData> grouped = groupedTimers.get(newTimers.getKey());
if (grouped == null) {
- grouped = new HashMap<>();
- groupedToMutate.put(newTimers.getKey(), grouped);
+ grouped = new ArrayList<>();
+ groupedTimers.put(newTimers.getKey(), grouped);
}
- grouped.put(newTimers.getValue().get(0).getDomain(), newTimers.getValue());
+ grouped.addAll(newTimers.getValue());
}
}
+ return groupedTimers;
}
private void updateTimers(TimerUpdate update) {
@@ -1334,24 +1340,35 @@ public class WatermarkManager {
* {@link WatermarkManager}.
*/
public static class FiredTimers {
- private final Map<TimeDomain, ? extends Collection<TimerData>> timers;
+ /** The transform the timers were set at and will be delivered to. */
+ private final AppliedPTransform<?, ?, ?> transform;
+ /** The key the timers were set for and will be delivered to. */
+ private final StructuralKey<?> key;
+ private final Collection<TimerData> timers;
- private FiredTimers(Map<TimeDomain, ? extends Collection<TimerData>> timers) {
+ private FiredTimers(
+ AppliedPTransform<?, ?, ?> transform, StructuralKey<?> key, Collection<TimerData> timers) {
+ this.transform = transform;
+ this.key = key;
this.timers = timers;
}
+ public AppliedPTransform<?, ?, ?> getTransform() {
+ return transform;
+ }
+
+ public StructuralKey<?> getKey() {
+ return key;
+ }
+
/**
* Gets all of the timers that have fired within the provided {@link TimeDomain}. If no timers
* fired within the provided domain, return an empty collection.
*
* <p>Timers within a {@link TimeDomain} are guaranteed to be in order of increasing timestamp.
*/
- public Collection<TimerData> getTimers(TimeDomain domain) {
- Collection<TimerData> domainTimers = timers.get(domain);
- if (domainTimers == null) {
- return Collections.emptyList();
- }
- return domainTimers;
+ public Collection<TimerData> getTimers() {
+ return timers;
}
@Override
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5dca2674/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java
index bc53570..e1277ac 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java
@@ -373,36 +373,31 @@ public class EvaluationContextTest {
.build();
// haven't added any timers, must be empty
- assertThat(context.extractFiredTimers().entrySet(), emptyIterable());
+ assertThat(context.extractFiredTimers(), emptyIterable());
context.handleResult(
context.createKeyedBundle(key, created).commit(Instant.now()),
ImmutableList.<TimerData>of(),
timerResult);
// timer hasn't fired
- assertThat(context.extractFiredTimers().entrySet(), emptyIterable());
+ assertThat(context.extractFiredTimers(), emptyIterable());
TransformResult advanceResult =
StepTransformResult.withoutHold(created.getProducingTransformInternal()).build();
// Should cause the downstream timer to fire
context.handleResult(null, ImmutableList.<TimerData>of(), advanceResult);
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> fired =
- context.extractFiredTimers();
+ Collection<FiredTimers> fired = context.extractFiredTimers();
assertThat(
- fired,
- Matchers.<AppliedPTransform<?, ?, ?>>hasKey(downstream.getProducingTransformInternal()));
- Map<StructuralKey<?>, FiredTimers> downstreamFired =
- fired.get(downstream.getProducingTransformInternal());
- assertThat(downstreamFired, Matchers.<Object>hasKey(key));
+ Iterables.getOnlyElement(fired).getKey(),
+ Matchers.<StructuralKey<?>>equalTo(key));
- FiredTimers firedForKey = downstreamFired.get(key);
- assertThat(firedForKey.getTimers(TimeDomain.PROCESSING_TIME), emptyIterable());
- assertThat(firedForKey.getTimers(TimeDomain.SYNCHRONIZED_PROCESSING_TIME), emptyIterable());
- assertThat(firedForKey.getTimers(TimeDomain.EVENT_TIME), contains(toFire));
+ FiredTimers firedForKey = Iterables.getOnlyElement(fired);
+ // Contains exclusively the fired timer
+ assertThat(firedForKey.getTimers(), contains(toFire));
// Don't reextract timers
- assertThat(context.extractFiredTimers().entrySet(), emptyIterable());
+ assertThat(context.extractFiredTimers(), emptyIterable());
}
@Test
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5dca2674/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
----------------------------------------------------------------------
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
index 1954005..6bde462 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
@@ -22,7 +22,6 @@ import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.not;
-import static org.hamcrest.Matchers.nullValue;
import static org.junit.Assert.assertThat;
import com.google.common.collect.ImmutableList;
@@ -68,6 +67,7 @@ import org.apache.beam.sdk.values.TimestampedValue;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
+import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.joda.time.ReadableInstant;
import org.junit.Before;
@@ -915,12 +915,9 @@ public class WatermarkManagerTest implements Serializable {
filteredDoubledWms.getSynchronizedProcessingOutputTime(),
not(earlierThan(initialFilteredDoubledWm)));
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> firedTimers =
- manager.extractFiredTimers();
+ Collection<FiredTimers> firedTimers = manager.extractFiredTimers();
assertThat(
- firedTimers.get(filtered.getProducingTransformInternal())
- .get(key)
- .getTimers(TimeDomain.PROCESSING_TIME),
+ Iterables.getOnlyElement(firedTimers).getTimers(),
contains(pastTimer));
// Our timer has fired, but has not been completed, so it holds our synchronized processing WM
assertThat(filteredWms.getSynchronizedProcessingOutputTime(), not(laterThan(startTime)));
@@ -1099,10 +1096,9 @@ public class WatermarkManagerTest implements Serializable {
@Test
public void extractFiredTimersReturnsFiredEventTimeTimers() {
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> initialTimers =
- manager.extractFiredTimers();
+ Collection<FiredTimers> initialTimers = manager.extractFiredTimers();
// Watermarks haven't advanced
- assertThat(initialTimers.entrySet(), emptyIterable());
+ assertThat(initialTimers, emptyIterable());
// Advance WM of keyed past the first timer, but ahead of the second and third
CommittedBundle<Integer> createdBundle = multiWindowedBundle(filtered);
@@ -1136,15 +1132,11 @@ public class WatermarkManagerTest implements Serializable {
new Instant(1000L));
manager.refreshAll();
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> firstTransformFiredTimers =
+ Collection<FiredTimers> firstFiredTimers =
manager.extractFiredTimers();
- assertThat(
- firstTransformFiredTimers.get(filtered.getProducingTransformInternal()), not(nullValue()));
- Map<StructuralKey<?>, FiredTimers> firstFilteredTimers =
- firstTransformFiredTimers.get(filtered.getProducingTransformInternal());
- assertThat(firstFilteredTimers.get(key), not(nullValue()));
- FiredTimers firstFired = firstFilteredTimers.get(key);
- assertThat(firstFired.getTimers(TimeDomain.EVENT_TIME), contains(earliestTimer));
+ assertThat(firstFiredTimers, not(Matchers.<FiredTimers>emptyIterable()));
+ FiredTimers firstFired = Iterables.getOnlyElement(firstFiredTimers);
+ assertThat(firstFired.getTimers(), contains(earliestTimer));
manager.updateWatermarks(null,
TimerUpdate.empty(),
@@ -1153,24 +1145,18 @@ public class WatermarkManagerTest implements Serializable {
Collections.<CommittedBundle<?>>emptyList()),
new Instant(50_000L));
manager.refreshAll();
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> secondTransformFiredTimers =
- manager.extractFiredTimers();
- assertThat(
- secondTransformFiredTimers.get(filtered.getProducingTransformInternal()), not(nullValue()));
- Map<StructuralKey<?>, FiredTimers> secondFilteredTimers =
- secondTransformFiredTimers.get(filtered.getProducingTransformInternal());
- assertThat(secondFilteredTimers.get(key), not(nullValue()));
- FiredTimers secondFired = secondFilteredTimers.get(key);
+ Collection<FiredTimers> secondFiredTimers = manager.extractFiredTimers();
+ assertThat(secondFiredTimers, not(Matchers.<FiredTimers>emptyIterable()));
+ FiredTimers secondFired = Iterables.getOnlyElement(secondFiredTimers);
// Contains, in order, middleTimer and then lastTimer
- assertThat(secondFired.getTimers(TimeDomain.EVENT_TIME), contains(middleTimer, lastTimer));
+ assertThat(secondFired.getTimers(), contains(middleTimer, lastTimer));
}
@Test
public void extractFiredTimersReturnsFiredProcessingTimeTimers() {
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> initialTimers =
- manager.extractFiredTimers();
+ Collection<FiredTimers> initialTimers = manager.extractFiredTimers();
// Watermarks haven't advanced
- assertThat(initialTimers.entrySet(), emptyIterable());
+ assertThat(initialTimers, emptyIterable());
// Advance WM of keyed past the first timer, but ahead of the second and third
CommittedBundle<Integer> createdBundle = multiWindowedBundle(filtered);
@@ -1204,15 +1190,10 @@ public class WatermarkManagerTest implements Serializable {
new Instant(1000L));
manager.refreshAll();
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> firstTransformFiredTimers =
- manager.extractFiredTimers();
- assertThat(
- firstTransformFiredTimers.get(filtered.getProducingTransformInternal()), not(nullValue()));
- Map<StructuralKey<?>, FiredTimers> firstFilteredTimers =
- firstTransformFiredTimers.get(filtered.getProducingTransformInternal());
- assertThat(firstFilteredTimers.get(key), not(nullValue()));
- FiredTimers firstFired = firstFilteredTimers.get(key);
- assertThat(firstFired.getTimers(TimeDomain.PROCESSING_TIME), contains(earliestTimer));
+ Collection<FiredTimers> firstFiredTimers = manager.extractFiredTimers();
+ assertThat(firstFiredTimers, not(Matchers.<FiredTimers>emptyIterable()));
+ FiredTimers firstFired = Iterables.getOnlyElement(firstFiredTimers);
+ assertThat(firstFired.getTimers(), contains(earliestTimer));
clock.set(new Instant(50_000L));
manager.updateWatermarks(null,
@@ -1222,24 +1203,19 @@ public class WatermarkManagerTest implements Serializable {
Collections.<CommittedBundle<?>>emptyList()),
new Instant(50_000L));
manager.refreshAll();
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> secondTransformFiredTimers =
+ Collection<FiredTimers> secondFiredTimers =
manager.extractFiredTimers();
- assertThat(
- secondTransformFiredTimers.get(filtered.getProducingTransformInternal()), not(nullValue()));
- Map<StructuralKey<?>, FiredTimers> secondFilteredTimers =
- secondTransformFiredTimers.get(filtered.getProducingTransformInternal());
- assertThat(secondFilteredTimers.get(key), not(nullValue()));
- FiredTimers secondFired = secondFilteredTimers.get(key);
+ assertThat(secondFiredTimers, not(Matchers.<FiredTimers>emptyIterable()));
+ FiredTimers secondFired = Iterables.getOnlyElement(secondFiredTimers);
// Contains, in order, middleTimer and then lastTimer
- assertThat(secondFired.getTimers(TimeDomain.PROCESSING_TIME), contains(middleTimer, lastTimer));
+ assertThat(secondFired.getTimers(), contains(middleTimer, lastTimer));
}
@Test
public void extractFiredTimersReturnsFiredSynchronizedProcessingTimeTimers() {
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> initialTimers =
- manager.extractFiredTimers();
+ Collection<FiredTimers> initialTimers = manager.extractFiredTimers();
// Watermarks haven't advanced
- assertThat(initialTimers.entrySet(), emptyIterable());
+ assertThat(initialTimers, emptyIterable());
// Advance WM of keyed past the first timer, but ahead of the second and third
CommittedBundle<Integer> createdBundle = multiWindowedBundle(filtered);
@@ -1273,16 +1249,11 @@ public class WatermarkManagerTest implements Serializable {
new Instant(1000L));
manager.refreshAll();
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> firstTransformFiredTimers =
+ Collection<FiredTimers> firstFiredTimers =
manager.extractFiredTimers();
- assertThat(
- firstTransformFiredTimers.get(filtered.getProducingTransformInternal()), not(nullValue()));
- Map<StructuralKey<?>, FiredTimers> firstFilteredTimers =
- firstTransformFiredTimers.get(filtered.getProducingTransformInternal());
- assertThat(firstFilteredTimers.get(key), not(nullValue()));
- FiredTimers firstFired = firstFilteredTimers.get(key);
- assertThat(
- firstFired.getTimers(TimeDomain.SYNCHRONIZED_PROCESSING_TIME), contains(earliestTimer));
+ assertThat(firstFiredTimers, not(Matchers.<FiredTimers>emptyIterable()));
+ FiredTimers firstFired = Iterables.getOnlyElement(firstFiredTimers);
+ assertThat(firstFired.getTimers(), contains(earliestTimer));
clock.set(new Instant(50_000L));
manager.updateWatermarks(null,
@@ -1292,18 +1263,11 @@ public class WatermarkManagerTest implements Serializable {
Collections.<CommittedBundle<?>>emptyList()),
new Instant(50_000L));
manager.refreshAll();
- Map<AppliedPTransform<?, ?, ?>, Map<StructuralKey<?>, FiredTimers>> secondTransformFiredTimers =
- manager.extractFiredTimers();
- assertThat(
- secondTransformFiredTimers.get(filtered.getProducingTransformInternal()), not(nullValue()));
- Map<StructuralKey<?>, FiredTimers> secondFilteredTimers =
- secondTransformFiredTimers.get(filtered.getProducingTransformInternal());
- assertThat(secondFilteredTimers.get(key), not(nullValue()));
- FiredTimers secondFired = secondFilteredTimers.get(key);
+ Collection<FiredTimers> secondFiredTimers = manager.extractFiredTimers();
+ assertThat(secondFiredTimers, not(Matchers.<FiredTimers>emptyIterable()));
+ FiredTimers secondFired = Iterables.getOnlyElement(secondFiredTimers);
// Contains, in order, middleTimer and then lastTimer
- assertThat(
- secondFired.getTimers(TimeDomain.SYNCHRONIZED_PROCESSING_TIME),
- contains(middleTimer, lastTimer));
+ assertThat(secondFired.getTimers(), contains(middleTimer, lastTimer));
}
@Test
[39/50] incubator-beam git commit: Enable javadoc for all modules by
default
Posted by th...@apache.org.
Enable javadoc for all modules by default
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/aae65db6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/aae65db6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/aae65db6
Branch: refs/heads/apex-runner
Commit: aae65db663c27de5b9a21adedc2110cc21cd4061
Parents: 14e093a
Author: Dan Halperin <dh...@google.com>
Authored: Fri Nov 4 15:28:43 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Fri Nov 4 18:29:21 2016 -0700
----------------------------------------------------------------------
examples/java/pom.xml | 68 ------------------------
examples/java8/pom.xml | 8 ---
pom.xml | 11 +++-
runners/core-java/pom.xml | 23 --------
runners/direct-java/pom.xml | 61 ---------------------
runners/flink/examples/pom.xml | 8 ---
runners/flink/runner/pom.xml | 8 ---
runners/google-cloud-dataflow-java/pom.xml | 70 -------------------------
runners/spark/pom.xml | 4 --
sdks/java/core/pom.xml | 70 -------------------------
sdks/java/io/jdbc/pom.xml | 4 --
sdks/java/io/kafka/pom.xml | 7 ---
sdks/java/io/kinesis/pom.xml | 7 ---
sdks/java/io/mongodb/pom.xml | 4 --
14 files changed, 10 insertions(+), 343 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/examples/java/pom.xml
----------------------------------------------------------------------
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 12a114f..857373a 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -285,74 +285,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <windowtitle>Apache Beam Examples</windowtitle>
- <doctitle>Apache Beam Examples</doctitle>
-
- <subpackages>org.apache.beam.examples</subpackages>
- <additionalparam>-exclude org.apache.beam.sdk.runners.worker:org.apache.beam.sdk.runners.dataflow:org.apache.beam.sdk.util ${beam.javadoc_opts}</additionalparam>
- <use>false</use>
- <quiet>true</quiet>
- <bottom><![CDATA[<br>]]></bottom>
-
- <offlineLinks>
- <!-- The Dataflow SDK docs -->
- <offlineLink>
- <url>https://cloud.google.com/dataflow/java-sdk/JavaDoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/dataflow-sdk-docs</location>
- </offlineLink>
- <!-- Other dependencies -->
- <offlineLink>
- <url>https://developers.google.com/api-client-library/java/google-api-java-client/reference/1.20.0/</url>
- <location>${basedir}/../../sdks/java/javadoc/apiclient-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://avro.apache.org/docs/${avro.version}/api/java/</url>
- <location>${basedir}/../../sdks/java/javadoc/avro-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/</url>
- <location>${basedir}/../../sdks/java/javadoc/bq-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://cloud.google.com/datastore/docs/apis/javadoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/datastore-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://docs.guava-libraries.googlecode.com/git-history/release18/javadoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/guava-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://fasterxml.github.io/jackson-annotations/javadoc/2.7/</url>
- <location>${basedir}/../../sdks/java/javadoc/jackson-annotations-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://fasterxml.github.io/jackson-databind/javadoc/2.7/</url>
- <location>${basedir}/../../sdks/java/javadoc/jackson-databind-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://www.joda.org/joda-time/apidocs</url>
- <location>${basedir}/../../sdks/java/javadoc/joda-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://developers.google.com/api-client-library/java/google-oauth-java-client/reference/1.20.0/</url>
- <location>${basedir}/../../sdks/java/javadoc/oauth-docs</location>
- </offlineLink>
- </offlineLinks>
- </configuration>
- <executions>
- <execution>
- <goals>
- <goal>jar</goal>
- </goals>
- <phase>package</phase>
- </execution>
- </executions>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/examples/java8/pom.xml
----------------------------------------------------------------------
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 18257d9..0a5dcb0 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -120,14 +120,6 @@
<build>
<plugins>
<plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <additionalparam>-Xdoclint:missing</additionalparam>
- </configuration>
- </plugin>
-
- <plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 14c38fd..77e87ba 100644
--- a/pom.xml
+++ b/pom.xml
@@ -160,6 +160,9 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
+ <configuration>
+ <additionalparam>${beam.javadoc_opts}</additionalparam>
+ </configuration>
<executions>
<execution>
<id>javadoc</id>
@@ -212,6 +215,11 @@
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
</plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-javadoc-plugin</artifactId>
+ </plugin>
</plugins>
</build>
</profile>
@@ -222,9 +230,10 @@
<jdk>[1.8,)</jdk>
</activation>
<properties>
- <beam.javadoc_opts>-Xdoclint:-missing</beam.javadoc_opts>
+ <beam.javadoc_opts>-Xdoclint:all -Xdoclint:-missing</beam.javadoc_opts>
</properties>
</profile>
+
<profile>
<id>java7-packageinfo</id>
<activation>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/runners/core-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index fa64660..3d8a84b 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -65,29 +65,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <windowtitle>Beam Runners Core utilities ${project.version} API</windowtitle>
- <doctitle>Beam Runners Core utilities for Java, version ${project.version}</doctitle>
- <overview>../javadoc/overview.html</overview>
-
- <subpackages>org.apache.beam.runners.core</subpackages>
- <use>false</use>
- <quiet>true</quiet>
- <bottom><![CDATA[<br>]]></bottom>
- </configuration>
- <executions>
- <execution>
- <goals>
- <goal>jar</goal>
- </goals>
- <phase>package</phase>
- </execution>
- </executions>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/runners/direct-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index b781c6e..64e9b5a 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -94,67 +94,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <windowtitle>Apache Beam Direct Runner ${project.version}</windowtitle>
- <doctitle>Apache Beam Direct Runner, version ${project.version}</doctitle>
-
- <subpackages>org.apache.beam.runners.direct</subpackages>
- <use>false</use>
- <quiet>true</quiet>
- <bottom><![CDATA[<br>]]></bottom>
-
- <offlineLinks>
- <offlineLink>
- <url>https://developers.google.com/api-client-library/java/google-api-java-client/reference/1.20.0/</url>
- <location>${basedir}/../../sdks/java/javadoc/apiclient-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://avro.apache.org/docs/${avro.version}/api/java/</url>
- <location>${basedir}/../../sdks/java/javadoc/avro-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/</url>
- <location>${basedir}/../../sdks/java/javadoc/bq-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://cloud.google.com/datastore/docs/apis/javadoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/datastore-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://docs.guava-libraries.googlecode.com/git-history/release19/javadoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/guava-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://hamcrest.org/JavaHamcrest/javadoc/1.3/</url>
- <location>${basedir}/../../sdks/java/javadoc/hamcrest-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://fasterxml.github.io/jackson-annotations/javadoc/2.7/</url>
- <location>${basedir}/../../sdks/java/javadoc/jackson-annotations-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://fasterxml.github.io/jackson-databind/javadoc/2.7/</url>
- <location>${basedir}/../../sdks/java/javadoc/jackson-databind-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://www.joda.org/joda-time/apidocs</url>
- <location>${basedir}/../../sdks/java/javadoc/joda-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://junit.sourceforge.net/javadoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/junit-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://developers.google.com/api-client-library/java/google-oauth-java-client/reference/1.20.0/</url>
- <location>${basedir}/../../sdks/java/javadoc/oauth-docs</location>
- </offlineLink>
- </offlineLinks>
- </configuration>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/runners/flink/examples/pom.xml
----------------------------------------------------------------------
diff --git a/runners/flink/examples/pom.xml b/runners/flink/examples/pom.xml
index 409d77b..051ec2d 100644
--- a/runners/flink/examples/pom.xml
+++ b/runners/flink/examples/pom.xml
@@ -110,14 +110,6 @@
</plugin>
<plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <additionalparam>-Xdoclint:missing</additionalparam>
- </configuration>
- </plugin>
-
- <plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<configuration>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/runners/flink/runner/pom.xml
----------------------------------------------------------------------
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 928889e..9729316 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -242,14 +242,6 @@
</executions>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <additionalparam>-Xdoclint:missing</additionalparam>
- </configuration>
- </plugin>
-
<!-- Integration Tests -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/runners/google-cloud-dataflow-java/pom.xml
----------------------------------------------------------------------
diff --git a/runners/google-cloud-dataflow-java/pom.xml b/runners/google-cloud-dataflow-java/pom.xml
index d8f1b82..0a2c9f6 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -85,76 +85,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <windowtitle>Google Cloud Dataflow Java Runner ${project.version}</windowtitle>
- <doctitle>Google Cloud Dataflow Runner for Java, version ${project.version}</doctitle>
-
- <subpackages>org.apache.beam.runners.dataflow</subpackages>
- <additionalparam>-exclude org.apache.beam.sdk.runners.dataflow.internal:org.apache.beam.sdk.runners.dataflow.testing:org.apache.beam.sdk.runners.dataflow.util ${beam.javadoc_opts}</additionalparam>
- <use>false</use>
- <quiet>true</quiet>
- <bottom><![CDATA[<br>]]></bottom>
-
- <offlineLinks>
- <offlineLink>
- <url>https://developers.google.com/api-client-library/java/google-api-java-client/reference/1.20.0/</url>
- <location>${basedir}/../../sdks/java/javadoc/apiclient-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://avro.apache.org/docs/${avro.version}/api/java/</url>
- <location>${basedir}/../../sdks/java/javadoc/avro-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/</url>
- <location>${basedir}/../../sdks/java/javadoc/bq-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://cloud.google.com/datastore/docs/apis/javadoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/datastore-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://docs.guava-libraries.googlecode.com/git-history/release19/javadoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/guava-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://hamcrest.org/JavaHamcrest/javadoc/1.3/</url>
- <location>${basedir}/../../sdks/java/javadoc/hamcrest-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://fasterxml.github.io/jackson-annotations/javadoc/2.7/</url>
- <location>${basedir}/../../sdks/java/javadoc/jackson-annotations-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://fasterxml.github.io/jackson-databind/javadoc/2.7/</url>
- <location>${basedir}/../../sdks/java/javadoc/jackson-databind-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://www.joda.org/joda-time/apidocs</url>
- <location>${basedir}/../../sdks/java/javadoc/joda-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://junit.sourceforge.net/javadoc/</url>
- <location>${basedir}/../../sdks/java/javadoc/junit-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://developers.google.com/api-client-library/java/google-oauth-java-client/reference/1.20.0/</url>
- <location>${basedir}/../../sdks/java/javadoc/oauth-docs</location>
- </offlineLink>
- </offlineLinks>
- </configuration>
- <executions>
- <execution>
- <goals>
- <goal>jar</goal>
- </goals>
- <phase>package</phase>
- </execution>
- </executions>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/runners/spark/pom.xml
----------------------------------------------------------------------
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 0edafe8..a3ca296 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -415,10 +415,6 @@
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
</plugin>
</plugins>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/sdks/java/core/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index 40a511c..7d4d9ab 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -44,71 +44,6 @@
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <windowtitle>Apache Beam SDK ${project.version} API</windowtitle>
- <doctitle>Apache Beam SDK for Java, version ${project.version}</doctitle>
- <overview>../javadoc/overview.html</overview>
-
- <subpackages>org.apache.beam.sdk</subpackages>
- <additionalparam>-exclude
- org.apache.beam.sdk.runners.worker:org.apache.beam.sdk.runners.dataflow:org.apache.beam.sdk.util:org.apache.beam.sdk.runners.inprocess
- ${beam.javadoc_opts}</additionalparam>
- <use>false</use>
- <quiet>true</quiet>
- <bottom><![CDATA[<br>]]></bottom>
-
- <offlineLinks>
- <offlineLink>
- <url>https://developers.google.com/api-client-library/java/google-api-java-client/reference/1.20.0/</url>
- <location>${basedir}/../javadoc/apiclient-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://avro.apache.org/docs/${avro.version}/api/java/</url>
- <location>${basedir}/../javadoc/avro-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/</url>
- <location>${basedir}/../javadoc/bq-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://cloud.google.com/datastore/docs/apis/javadoc/</url>
- <location>${basedir}/../javadoc/datastore-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://docs.guava-libraries.googlecode.com/git-history/release19/javadoc/</url>
- <location>${basedir}/../javadoc/guava-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://hamcrest.org/JavaHamcrest/javadoc/1.3/</url>
- <location>${basedir}/../javadoc/hamcrest-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://fasterxml.github.io/jackson-annotations/javadoc/2.7/</url>
- <location>${basedir}/../javadoc/jackson-annotations-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://fasterxml.github.io/jackson-databind/javadoc/2.7/</url>
- <location>${basedir}/../javadoc/jackson-databind-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://www.joda.org/joda-time/apidocs</url>
- <location>${basedir}/../javadoc/joda-docs</location>
- </offlineLink>
- <offlineLink>
- <url>http://junit.sourceforge.net/javadoc/</url>
- <location>${basedir}/../javadoc/junit-docs</location>
- </offlineLink>
- <offlineLink>
- <url>https://developers.google.com/api-client-library/java/google-oauth-java-client/reference/1.20.0/</url>
- <location>${basedir}/../javadoc/oauth-docs</location>
- </offlineLink>
- </offlineLinks>
- </configuration>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<!-- Set testSourceDirectory in order to exclude generated-test-sources -->
@@ -172,11 +107,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/sdks/java/io/jdbc/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/pom.xml b/sdks/java/io/jdbc/pom.xml
index a65ed46..2eb24bd 100644
--- a/sdks/java/io/jdbc/pom.xml
+++ b/sdks/java/io/jdbc/pom.xml
@@ -48,10 +48,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- </plugin>
</plugins>
</build>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/sdks/java/io/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/kafka/pom.xml b/sdks/java/io/kafka/pom.xml
index 184c1c9..2004952 100644
--- a/sdks/java/io/kafka/pom.xml
+++ b/sdks/java/io/kafka/pom.xml
@@ -52,13 +52,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <additionalparam>-Xdoclint:missing</additionalparam>
- </configuration>
- </plugin>
</plugins>
</build>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/sdks/java/io/kinesis/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/kinesis/pom.xml b/sdks/java/io/kinesis/pom.xml
index 76df329..439995d 100644
--- a/sdks/java/io/kinesis/pom.xml
+++ b/sdks/java/io/kinesis/pom.xml
@@ -52,13 +52,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- <configuration>
- <additionalparam>-Xdoclint:missing</additionalparam>
- </configuration>
- </plugin>
<!-- Integration Tests -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/aae65db6/sdks/java/io/mongodb/pom.xml
----------------------------------------------------------------------
diff --git a/sdks/java/io/mongodb/pom.xml b/sdks/java/io/mongodb/pom.xml
index d674e63..d0ae6ac 100644
--- a/sdks/java/io/mongodb/pom.xml
+++ b/sdks/java/io/mongodb/pom.xml
@@ -48,10 +48,6 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-javadoc-plugin</artifactId>
- </plugin>
</plugins>
</build>
[13/50] incubator-beam git commit: Closes #1183
Posted by th...@apache.org.
Closes #1183
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c6c41eac
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c6c41eac
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c6c41eac
Branch: refs/heads/apex-runner
Commit: c6c41eac3dca8508ab7920a66f3298fd3fd3f942
Parents: bdd71c4 d6f3a02
Author: Dan Halperin <dh...@google.com>
Authored: Thu Nov 3 12:31:48 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Thu Nov 3 12:31:48 2016 -0700
----------------------------------------------------------------------
.../apache/beam/sdk/io/gcp/ApiSurfaceTest.java | 132 +++++++++++++++++++
1 file changed, 132 insertions(+)
----------------------------------------------------------------------
[47/50] incubator-beam git commit: Format Regex according to style
guidelines
Posted by th...@apache.org.
Format Regex according to style guidelines
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/79b04551
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/79b04551
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/79b04551
Branch: refs/heads/apex-runner
Commit: 79b04551c7c9f964908ab4a1d95119ef8a7fff84
Parents: 6954abe
Author: Kenneth Knowles <kl...@google.com>
Authored: Mon Nov 7 10:10:59 2016 -0800
Committer: Kenneth Knowles <kl...@google.com>
Committed: Mon Nov 7 10:10:59 2016 -0800
----------------------------------------------------------------------
.../org/apache/beam/sdk/transforms/Regex.java | 525 +++++++++----------
.../apache/beam/sdk/transforms/RegexTest.java | 106 ++--
2 files changed, 283 insertions(+), 348 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/79b04551/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java
index 27104f6..a94130d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java
@@ -24,22 +24,17 @@ import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
/**
- * {@code PTransorm}s to use Regular Expressions to process elements in a
- * {@link PCollection}.
+ * {@code PTransorm}s to use Regular Expressions to process elements in a {@link PCollection}.
*
- * <p>
- * {@link Regex#matches(String, int)} can be used to see if an entire line matches
- * a Regex. {@link Regex#matchesKV(String, int, int)} can be used to see if an entire
- * line matches a Regex and output certain groups as a {@link KV}.
- * </p>
- * <p>
- * {@link Regex#find(String, int)} can be used to see if a portion of a line
- * matches a Regex. {@link Regex#matchesKV(String, int, int)} can be used to see if a
- * portion of a line matches a Regex and output certain groups as a {@link KV}.
- * </p>
- * <p>
- * Lines that do not match the Regex will not be output.
- * </p>
+ * <p>{@link Regex#matches(String, int)} can be used to see if an entire line matches a Regex.
+ * {@link Regex#matchesKV(String, int, int)} can be used to see if an entire line matches a Regex
+ * and output certain groups as a {@link KV}.
+ *
+ * <p>{@link Regex#find(String, int)} can be used to see if a portion of a line matches a Regex.
+ * {@link Regex#matchesKV(String, int, int)} can be used to see if a portion of a line matches a
+ * Regex and output certain groups as a {@link KV}.
+ *
+ * <p>Lines that do not match the Regex will not be output.
*/
public class Regex {
private Regex() {
@@ -47,159 +42,135 @@ public class Regex {
}
/**
- * Returns a {@link Regex.Matches} {@link PTransform} that checks if
- * the entire line matches the Regex. Returns the entire line (group 0) as a
- * {@link PCollection}.
- * @param regex
- * The regular expression to run
+ * Returns a {@link Regex.Matches} {@link PTransform} that checks if the entire line matches the
+ * Regex. Returns the entire line (group 0) as a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
*/
public static Matches matches(String regex) {
return matches(regex, 0);
}
/**
- * Returns a {@link Regex.Matches} {@link PTransform} that checks if
- * the entire line matches the Regex. Returns the group as a
- * {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param group
- * The Regex group to return as a PCollection
+ * Returns a {@link Regex.Matches} {@link PTransform} that checks if the entire line matches the
+ * Regex. Returns the group as a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
+ * @param group The Regex group to return as a PCollection
*/
public static Matches matches(String regex, int group) {
return new Matches(regex, group);
}
/**
- * Returns a {@link Regex.MatchesKV} {@link PTransform} that checks
- * if the entire line matches the Regex. Returns the specified groups as the
- * key and value as a {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param keyGroup
- * The Regex group to use as the key
- * @param valueGroup
- * The Regex group to use the value
+ * Returns a {@link Regex.MatchesKV} {@link PTransform} that checks if the entire line matches the
+ * Regex. Returns the specified groups as the key and value as a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
+ * @param keyGroup The Regex group to use as the key
+ * @param valueGroup The Regex group to use the value
*/
- public static MatchesKV matchesKV(String regex, int keyGroup,
- int valueGroup) {
+ public static MatchesKV matchesKV(String regex, int keyGroup, int valueGroup) {
return new MatchesKV(regex, keyGroup, valueGroup);
}
/**
- * Returns a {@link Regex.Find} {@link PTransform} that checks if a
- * portion of the line matches the Regex. Returns the entire line (group 0) as
- * a {@link PCollection}.
- * @param regex
- * The regular expression to run
+ * Returns a {@link Regex.Find} {@link PTransform} that checks if a portion of the line matches
+ * the Regex. Returns the entire line (group 0) as a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
*/
public static Find find(String regex) {
return find(regex, 0);
}
/**
- * Returns a {@link Regex.Find} {@link PTransform} that checks if a
- * portion of the line matches the Regex. Returns the group as a
- * {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param group
- * The Regex group to return as a PCollection
+ * Returns a {@link Regex.Find} {@link PTransform} that checks if a portion of the line matches
+ * the Regex. Returns the group as a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
+ * @param group The Regex group to return as a PCollection
*/
public static Find find(String regex, int group) {
return new Find(regex, group);
}
/**
- * Returns a {@link Regex.FindKV} {@link PTransform} that checks if a
- * portion of the line matches the Regex. Returns the specified groups as the
- * key and value as a {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param keyGroup
- * The Regex group to use as the key
- * @param valueGroup
- * The Regex group to use the value
+ * Returns a {@link Regex.FindKV} {@link PTransform} that checks if a portion of the line matches
+ * the Regex. Returns the specified groups as the key and value as a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
+ * @param keyGroup The Regex group to use as the key
+ * @param valueGroup The Regex group to use the value
*/
public static FindKV findKV(String regex, int keyGroup, int valueGroup) {
return new FindKV(regex, keyGroup, valueGroup);
}
/**
- * Returns a {@link Regex.ReplaceAll} {@link PTransform} that checks if a
- * portion of the line matches the Regex and replaces all matches with the replacement
- * String. Returns the group as a {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param replacement
- * The string to be substituted for each match
+ * Returns a {@link Regex.ReplaceAll} {@link PTransform} that checks if a portion of the line
+ * matches the Regex and replaces all matches with the replacement String. Returns the group as a
+ * {@link PCollection}.
+ *
+ * @param regex The regular expression to run
+ * @param replacement The string to be substituted for each match
*/
public static ReplaceAll replaceAll(String regex, String replacement) {
return new ReplaceAll(regex, replacement);
}
/**
- * Returns a {@link Regex.ReplaceAll} {@link PTransform} that checks if a
- * portion of the line matches the Regex and replaces the first match with the replacement
- * String. Returns the group as a {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param replacement
- * The string to be substituted for each match
+ * Returns a {@link Regex.ReplaceAll} {@link PTransform} that checks if a portion of the line
+ * matches the Regex and replaces the first match with the replacement String. Returns the group
+ * as a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
+ * @param replacement The string to be substituted for each match
*/
public static ReplaceFirst replaceFirst(String regex, String replacement) {
return new ReplaceFirst(regex, replacement);
}
- /**
- * Returns a {@link Regex.Split} {@link PTransform} that splits a string
- * on the regular expression and then outputs each item. It will not output empty
- * items. Returns the group as a {@link PCollection}.
- * a {@link PCollection}.
- * @param regex
- * The regular expression to run
+ /**
+ * Returns a {@link Regex.Split} {@link PTransform} that splits a string on the regular expression
+ * and then outputs each item. It will not output empty items. Returns the group as a {@link
+ * PCollection}. a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
*/
public static Split split(String regex) {
return split(regex, false);
}
/**
- * Returns a {@link Regex.Split} {@link PTransform} that splits a string
- * on the regular expression and then outputs each item. Returns the group as a
- * {@link PCollection}.
- * @param regex
- * The regular expression to run
- * @param outputEmpty
- * Should empty be output. True to output empties and false if not.
+ * Returns a {@link Regex.Split} {@link PTransform} that splits a string on the regular expression
+ * and then outputs each item. Returns the group as a {@link PCollection}.
+ *
+ * @param regex The regular expression to run
+ * @param outputEmpty Should empty be output. True to output empties and false if not.
*/
public static Split split(String regex, boolean outputEmpty) {
return new Split(regex, outputEmpty);
}
/**
- * {@code Regex.Matches<String>} takes a {@code PCollection<String>}
- * and returns a {@code PCollection<String>} representing the value
- * extracted from the Regex groups of the input {@code PCollection}
- * to the number of times that element occurs in the input.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If the entire line
- * does not match the Regex, the line will not be output. If it does match the
- * entire line, the group in the Regex will be used. The output will be the
- * Regex group.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
+ * {@code Regex.Matches<String>} takes a {@code PCollection<String>} and returns a {@code
+ * PCollection<String>} representing the value extracted from the Regex groups of the input {@code
+ * PCollection} to the number of times that element occurs in the input.
+ *
+ * <p>This transform runs a Regex on the entire input line. If the entire line does not match the
+ * Regex, the line will not be output. If it does match the entire line, the group in the Regex
+ * will be used. The output will be the Regex group.
+ *
+ * <p>Example of use:
+ *
+ * <pre>{@code
* PCollection<String> words = ...;
* PCollection<String> values =
* words.apply(Regex.matches("myregex (mygroup)", 1));
- * }
- * </pre>
+ * }</pre>
*/
- public static class Matches
- extends PTransform<PCollection<String>, PCollection<String>> {
+ public static class Matches extends PTransform<PCollection<String>, PCollection<String>> {
Pattern pattern;
int group;
@@ -209,42 +180,38 @@ public class Regex {
}
public PCollection<String> apply(PCollection<String> in) {
- return in
- .apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
-
- if (m.matches()) {
- c.output(m.group(group));
- }
- }
- }));
+ return in.apply(
+ ParDo.of(
+ new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.matches()) {
+ c.output(m.group(group));
+ }
+ }
+ }));
}
}
/**
- * {@code Regex.MatchesKV<KV<String, String>>} takes a
- * {@code PCollection<String>} and returns a
- * {@code PCollection<KV<String, String>>} representing the key and value
- * extracted from the Regex groups of the input {@code PCollection} to the
- * number of times that element occurs in the input.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If the entire line
- * does not match the Regex, the line will not be output. If it does match the
- * entire line, the groups in the Regex will be used. The key will be the
- * key's group and the value will be the value's group.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
+ * {@code Regex.MatchesKV<KV<String, String>>} takes a {@code PCollection<String>} and returns a
+ * {@code PCollection<KV<String, String>>} representing the key and value extracted from the Regex
+ * groups of the input {@code PCollection} to the number of times that element occurs in the
+ * input.
+ *
+ * <p>This transform runs a Regex on the entire input line. If the entire line does not match the
+ * Regex, the line will not be output. If it does match the entire line, the groups in the Regex
+ * will be used. The key will be the key's group and the value will be the value's group.
+ *
+ * <p>Example of use:
+ *
+ * <pre>{@code
* PCollection<String> words = ...;
* PCollection<KV<String, String>> keysAndValues =
* words.apply(Regex.matchesKV("myregex (mykeygroup) (myvaluegroup)", 1, 2));
- * }
- * </pre>
+ * }</pre>
*/
public static class MatchesKV
extends PTransform<PCollection<String>, PCollection<KV<String, String>>> {
@@ -258,44 +225,39 @@ public class Regex {
}
public PCollection<KV<String, String>> apply(PCollection<String> in) {
- return in.apply(ParDo
- .of(new DoFn<String, KV<String, String>>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
-
- if (m.find()) {
- c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
- }
- }
- }));
+ return in.apply(
+ ParDo.of(
+ new DoFn<String, KV<String, String>>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
+ }
+ }
+ }));
}
}
/**
- * {@code Regex.Find<String>} takes a {@code PCollection<String>} and
- * returns a {@code PCollection<String>} representing the value extracted
- * from the Regex groups of the input {@code PCollection} to
- * the number of times that element occurs in the input.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If a portion of the
- * line does not match the Regex, the line will not be output. If it does
- * match a portion of the line, the group in the Regex will be used. The
- * output will be the Regex group.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
+ * {@code Regex.Find<String>} takes a {@code PCollection<String>} and returns a {@code
+ * PCollection<String>} representing the value extracted from the Regex groups of the input {@code
+ * PCollection} to the number of times that element occurs in the input.
+ *
+ * <p>This transform runs a Regex on the entire input line. If a portion of the line does not
+ * match the Regex, the line will not be output. If it does match a portion of the line, the group
+ * in the Regex will be used. The output will be the Regex group.
+ *
+ * <p>Example of use:
+ *
+ * <pre>{@code
* PCollection<String> words = ...;
* PCollection<String> values =
* words.apply(Regex.find("myregex (mygroup)", 1));
- * }
- * </pre>
+ * }</pre>
*/
- public static class Find
- extends PTransform<PCollection<String>, PCollection<String>> {
+ public static class Find extends PTransform<PCollection<String>, PCollection<String>> {
Pattern pattern;
int group;
@@ -305,41 +267,39 @@ public class Regex {
}
public PCollection<String> apply(PCollection<String> in) {
- return in.apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
-
- if (m.find()) {
- c.output(m.group(group));
- }
- }
- }));
+ return in.apply(
+ ParDo.of(
+ new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(m.group(group));
+ }
+ }
+ }));
}
}
/**
- * {@code Regex.MatchesKV<KV<String, String>>} takes a
- * {@code PCollection<String>} and returns a
- * {@code PCollection<KV<String, String>>} representing the key and value
- * extracted from the Regex groups of the input {@code PCollection} to the
- * number of times that element occurs in the input.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If a portion of the
- * line does not match the Regex, the line will not be output. If it does
- * match a portion of the line, the groups in the Regex will be used. The key
- * will be the key's group and the value will be the value's group.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
+ * {@code Regex.MatchesKV<KV<String, String>>} takes a {@code PCollection<String>} and returns a
+ * {@code PCollection<KV<String, String>>} representing the key and value extracted from the Regex
+ * groups of the input {@code PCollection} to the number of times that element occurs in the
+ * input.
+ *
+ * <p>This transform runs a Regex on the entire input line. If a portion of the line does not
+ * match the Regex, the line will not be output. If it does match a portion of the line, the
+ * groups in the Regex will be used. The key will be the key's group and the value will be the
+ * value's group.
+ *
+ * <p>Example of use:
+ *
+ * <pre>{@code
* PCollection<String> words = ...;
* PCollection<KV<String, String>> keysAndValues =
* words.apply(Regex.findKV("myregex (mykeygroup) (myvaluegroup)", 1, 2));
- * }
- * </pre>
+ * }</pre>
*/
public static class FindKV
extends PTransform<PCollection<String>, PCollection<KV<String, String>>> {
@@ -354,42 +314,38 @@ public class Regex {
public PCollection<KV<String, String>> apply(PCollection<String> in) {
return in.apply(
- ParDo.of(new DoFn<String, KV<String, String>>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
-
- if (m.find()) {
- c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
- }
- }
- }));
+ ParDo.of(
+ new DoFn<String, KV<String, String>>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+
+ if (m.find()) {
+ c.output(KV.of(m.group(keyGroup), m.group(valueGroup)));
+ }
+ }
+ }));
}
}
/**
- * {@code Regex.ReplaceAll<String>} takes a {@code PCollection<String>} and
- * returns a {@code PCollection<String>} with all Strings that matched the
- * Regex being replaced with the replacement string.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If a portion of the
- * line does not match the Regex, the line will be output without changes. If it does
- * match a portion of the line, all portions matching the Regex will be replaced
- * with the replacement String.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
+ * {@code Regex.ReplaceAll<String>} takes a {@code PCollection<String>} and returns a {@code
+ * PCollection<String>} with all Strings that matched the Regex being replaced with the
+ * replacement string.
+ *
+ * <p>This transform runs a Regex on the entire input line. If a portion of the line does not
+ * match the Regex, the line will be output without changes. If it does match a portion of the
+ * line, all portions matching the Regex will be replaced with the replacement String.
+ *
+ * <p>Example of use:
+ *
+ * <pre>{@code
* PCollection<String> words = ...;
* PCollection<String> values =
* words.apply(Regex.replaceAll("myregex", "myreplacement"));
- * }
- * </pre>
+ * }</pre>
*/
- public static class ReplaceAll
- extends PTransform<PCollection<String>, PCollection<String>> {
+ public static class ReplaceAll extends PTransform<PCollection<String>, PCollection<String>> {
Pattern pattern;
String replacement;
@@ -399,39 +355,36 @@ public class Regex {
}
public PCollection<String> apply(PCollection<String> in) {
- return in.apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
- c.output(m.replaceAll(replacement));
- }
- }));
+ return in.apply(
+ ParDo.of(
+ new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+ c.output(m.replaceAll(replacement));
+ }
+ }));
}
}
/**
- * {@code Regex.ReplaceFirst<String>} takes a {@code PCollection<String>} and
- * returns a {@code PCollection<String>} with the first Strings that matched the
- * Regex being replaced with the replacement string.
- *
- * <p>
- * This transform runs a Regex on the entire input line. If a portion of the
- * line does not match the Regex, the line will be output without changes. If it does
- * match a portion of the line, the first portion matching the Regex will be replaced
- * with the replacement String.
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
+ * {@code Regex.ReplaceFirst<String>} takes a {@code PCollection<String>} and returns a {@code
+ * PCollection<String>} with the first Strings that matched the Regex being replaced with the
+ * replacement string.
+ *
+ * <p>This transform runs a Regex on the entire input line. If a portion of the line does not
+ * match the Regex, the line will be output without changes. If it does match a portion of the
+ * line, the first portion matching the Regex will be replaced with the replacement String.
+ *
+ * <p>Example of use:
+ *
+ * <pre>{@code
* PCollection<String> words = ...;
* PCollection<String> values =
* words.apply(Regex.replaceFirst("myregex", "myreplacement"));
- * }
- * </pre>
+ * }</pre>
*/
- public static class ReplaceFirst
- extends PTransform<PCollection<String>, PCollection<String>> {
+ public static class ReplaceFirst extends PTransform<PCollection<String>, PCollection<String>> {
Pattern pattern;
String replacement;
@@ -441,44 +394,38 @@ public class Regex {
}
public PCollection<String> apply(PCollection<String> in) {
- return in.apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- Matcher m = pattern.matcher((String) c.element());
- c.output(m.replaceFirst(replacement));
- }
- }));
+ return in.apply(
+ ParDo.of(
+ new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ Matcher m = pattern.matcher((String) c.element());
+ c.output(m.replaceFirst(replacement));
+ }
+ }));
}
}
/**
- * {@code Regex.Split<String>} takes a {@code PCollection<String>} and
- * returns a {@code PCollection<String>} with the input string split into
- * individual items in a list. Each item is then output as a separate string.
- *
- * <p>
- * This transform runs a Regex as part of a splint the entire input line. The split
- * gives back an array of items. Each item is output as a separate item in the
- * {@code PCollection<String>}.
- * </p>
- *
- * <p>
- * Depending on the Regex, a split can be an empty or
- * "" string. You can pass in a parameter if you want empty strings or not.
- * </p>
- *
- * <p>
- * Example of use:
- * <pre>
- * {@code
+ * {@code Regex.Split<String>} takes a {@code PCollection<String>} and returns a {@code
+ * PCollection<String>} with the input string split into individual items in a list. Each item is
+ * then output as a separate string.
+ *
+ * <p>This transform runs a Regex as part of a splint the entire input line. The split gives back
+ * an array of items. Each item is output as a separate item in the {@code PCollection<String>}.
+ *
+ * <p>Depending on the Regex, a split can be an empty or "" string. You can pass in a parameter if
+ * you want empty strings or not.
+ *
+ * <p>Example of use:
+ *
+ * <pre>{@code
* PCollection<String> words = ...;
* PCollection<String> values =
* words.apply(Regex.split("\W*"));
- * }
- * </pre>
+ * }</pre>
*/
- public static class Split
- extends PTransform<PCollection<String>, PCollection<String>> {
+ public static class Split extends PTransform<PCollection<String>, PCollection<String>> {
Pattern pattern;
boolean outputEmpty;
@@ -488,18 +435,20 @@ public class Regex {
}
public PCollection<String> apply(PCollection<String> in) {
- return in.apply(ParDo.of(new DoFn<String, String>() {
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- String[] items = pattern.split(c.element());
-
- for (String item : items) {
- if (outputEmpty || !item.isEmpty()) {
- c.output(item);
- }
- }
- }
- }));
+ return in.apply(
+ ParDo.of(
+ new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ String[] items = pattern.split(c.element());
+
+ for (String item : items) {
+ if (outputEmpty || !item.isEmpty()) {
+ c.output(item);
+ }
+ }
+ }
+ }));
}
}
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/79b04551/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java
index 71f080e..6e196b4 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RegexTest.java
@@ -30,9 +30,7 @@ import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-/**
- * Tests for {@link Regex}.
- */
+/** Tests for {@link Regex}. */
@RunWith(JUnit4.class)
public class RegexTest implements Serializable {
@Test
@@ -40,9 +38,8 @@ public class RegexTest implements Serializable {
public void testFind() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("aj", "xj", "yj", "zj"))
- .apply(Regex.find("[xyz]"));
+ PCollection<String> output =
+ p.apply(Create.of("aj", "xj", "yj", "zj")).apply(Regex.find("[xyz]"));
PAssert.that(output).containsInAnyOrder("x", "y", "z");
p.run();
@@ -53,9 +50,8 @@ public class RegexTest implements Serializable {
public void testFindGroup() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("aj", "xj", "yj", "zj"))
- .apply(Regex.find("([xyz])", 1));
+ PCollection<String> output =
+ p.apply(Create.of("aj", "xj", "yj", "zj")).apply(Regex.find("([xyz])", 1));
PAssert.that(output).containsInAnyOrder("x", "y", "z");
p.run();
@@ -66,9 +62,7 @@ public class RegexTest implements Serializable {
public void testFindNone() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("a", "b", "c", "d"))
- .apply(Regex.find("[xyz]"));
+ PCollection<String> output = p.apply(Create.of("a", "b", "c", "d")).apply(Regex.find("[xyz]"));
PAssert.that(output).empty();
p.run();
@@ -79,9 +73,8 @@ public class RegexTest implements Serializable {
public void testKVFind() {
TestPipeline p = TestPipeline.create();
- PCollection<KV<String, String>> output = p
- .apply(Create.of("a b c"))
- .apply(Regex.findKV("a (b) (c)", 1, 2));
+ PCollection<KV<String, String>> output =
+ p.apply(Create.of("a b c")).apply(Regex.findKV("a (b) (c)", 1, 2));
PAssert.that(output).containsInAnyOrder(KV.of("b", "c"));
p.run();
@@ -92,9 +85,8 @@ public class RegexTest implements Serializable {
public void testKVFindNone() {
TestPipeline p = TestPipeline.create();
- PCollection<KV<String, String>> output = p
- .apply(Create.of("x y z"))
- .apply(Regex.findKV("a (b) (c)", 1, 2));
+ PCollection<KV<String, String>> output =
+ p.apply(Create.of("x y z")).apply(Regex.findKV("a (b) (c)", 1, 2));
PAssert.that(output).empty();
p.run();
@@ -105,9 +97,8 @@ public class RegexTest implements Serializable {
public void testMatches() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("a", "x", "y", "z"))
- .apply(Regex.matches("[xyz]"));
+ PCollection<String> output =
+ p.apply(Create.of("a", "x", "y", "z")).apply(Regex.matches("[xyz]"));
PAssert.that(output).containsInAnyOrder("x", "y", "z");
p.run();
@@ -118,9 +109,8 @@ public class RegexTest implements Serializable {
public void testMatchesNone() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("a", "b", "c", "d"))
- .apply(Regex.matches("[xyz]"));
+ PCollection<String> output =
+ p.apply(Create.of("a", "b", "c", "d")).apply(Regex.matches("[xyz]"));
PAssert.that(output).empty();
p.run();
@@ -131,9 +121,8 @@ public class RegexTest implements Serializable {
public void testMatchesGroup() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("a", "x xxx", "x yyy", "x zzz"))
- .apply(Regex.matches("x ([xyz]*)", 1));
+ PCollection<String> output =
+ p.apply(Create.of("a", "x xxx", "x yyy", "x zzz")).apply(Regex.matches("x ([xyz]*)", 1));
PAssert.that(output).containsInAnyOrder("xxx", "yyy", "zzz");
p.run();
@@ -144,9 +133,8 @@ public class RegexTest implements Serializable {
public void testKVMatches() {
TestPipeline p = TestPipeline.create();
- PCollection<KV<String, String>> output = p
- .apply(Create.of("a b c"))
- .apply(Regex.matchesKV("a (b) (c)", 1, 2));
+ PCollection<KV<String, String>> output =
+ p.apply(Create.of("a b c")).apply(Regex.matchesKV("a (b) (c)", 1, 2));
PAssert.that(output).containsInAnyOrder(KV.of("b", "c"));
p.run();
@@ -157,9 +145,8 @@ public class RegexTest implements Serializable {
public void testKVMatchesNone() {
TestPipeline p = TestPipeline.create();
- PCollection<KV<String, String>> output = p
- .apply(Create.of("x y z"))
- .apply(Regex.matchesKV("a (b) (c)", 1, 2));
+ PCollection<KV<String, String>> output =
+ p.apply(Create.of("x y z")).apply(Regex.matchesKV("a (b) (c)", 1, 2));
PAssert.that(output).empty();
p.run();
}
@@ -169,9 +156,8 @@ public class RegexTest implements Serializable {
public void testReplaceAll() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("xj", "yj", "zj"))
- .apply(Regex.replaceAll("[xyz]", "new"));
+ PCollection<String> output =
+ p.apply(Create.of("xj", "yj", "zj")).apply(Regex.replaceAll("[xyz]", "new"));
PAssert.that(output).containsInAnyOrder("newj", "newj", "newj");
p.run();
@@ -182,9 +168,8 @@ public class RegexTest implements Serializable {
public void testReplaceAllMixed() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("abc", "xj", "yj", "zj", "def"))
- .apply(Regex.replaceAll("[xyz]", "new"));
+ PCollection<String> output =
+ p.apply(Create.of("abc", "xj", "yj", "zj", "def")).apply(Regex.replaceAll("[xyz]", "new"));
PAssert.that(output).containsInAnyOrder("abc", "newj", "newj", "newj", "def");
p.run();
@@ -195,9 +180,8 @@ public class RegexTest implements Serializable {
public void testReplaceFirst() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("xjx", "yjy", "zjz"))
- .apply(Regex.replaceFirst("[xyz]", "new"));
+ PCollection<String> output =
+ p.apply(Create.of("xjx", "yjy", "zjz")).apply(Regex.replaceFirst("[xyz]", "new"));
PAssert.that(output).containsInAnyOrder("newjx", "newjy", "newjz");
p.run();
@@ -208,9 +192,9 @@ public class RegexTest implements Serializable {
public void testReplaceFirstMixed() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("abc", "xjx", "yjy", "zjz", "def"))
- .apply(Regex.replaceFirst("[xyz]", "new"));
+ PCollection<String> output =
+ p.apply(Create.of("abc", "xjx", "yjy", "zjz", "def"))
+ .apply(Regex.replaceFirst("[xyz]", "new"));
PAssert.that(output).containsInAnyOrder("abc", "newjx", "newjy", "newjz", "def");
p.run();
@@ -221,12 +205,12 @@ public class RegexTest implements Serializable {
public void testSplits() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("The quick brown fox jumps over the lazy dog"))
- .apply(Regex.split("\\W+"));
+ PCollection<String> output =
+ p.apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(Regex.split("\\W+"));
- PAssert.that(output).containsInAnyOrder("The", "quick", "brown",
- "fox", "jumps", "over", "the", "lazy", "dog");
+ PAssert.that(output)
+ .containsInAnyOrder("The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog");
p.run();
}
@@ -235,14 +219,16 @@ public class RegexTest implements Serializable {
public void testSplitsWithEmpty() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("The quick brown fox jumps over the lazy dog"))
- .apply(Regex.split("\\s", true));
+ PCollection<String> output =
+ p.apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(Regex.split("\\s", true));
String[] outputStr = "The quick brown fox jumps over the lazy dog".split("\\s");
- PAssert.that(output).containsInAnyOrder("The", "", "quick", "brown", "", "",
- "fox", "jumps", "over", "", "", "", "the", "lazy", "dog");
+ PAssert.that(output)
+ .containsInAnyOrder(
+ "The", "", "quick", "brown", "", "", "fox", "jumps", "over", "", "", "", "the", "lazy",
+ "dog");
p.run();
}
@@ -251,12 +237,12 @@ public class RegexTest implements Serializable {
public void testSplitsWithoutEmpty() {
TestPipeline p = TestPipeline.create();
- PCollection<String> output = p
- .apply(Create.of("The quick brown fox jumps over the lazy dog"))
- .apply(Regex.split("\\s", false));
+ PCollection<String> output =
+ p.apply(Create.of("The quick brown fox jumps over the lazy dog"))
+ .apply(Regex.split("\\s", false));
- PAssert.that(output).containsInAnyOrder("The", "quick", "brown",
- "fox", "jumps", "over", "the", "lazy", "dog");
+ PAssert.that(output)
+ .containsInAnyOrder("The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog");
p.run();
}
}
[05/50] incubator-beam git commit: Closes #1263
Posted by th...@apache.org.
Closes #1263
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4dec38ce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4dec38ce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4dec38ce
Branch: refs/heads/apex-runner
Commit: 4dec38cee290335eeac6ba131386dd7d27c8d85d
Parents: 5224693 2f6104c
Author: Dan Halperin <dh...@google.com>
Authored: Wed Nov 2 15:53:26 2016 -0700
Committer: Dan Halperin <dh...@google.com>
Committed: Wed Nov 2 15:53:26 2016 -0700
----------------------------------------------------------------------
sdks/java/core/src/main/java/org/apache/beam/sdk/io/Sink.java | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
----------------------------------------------------------------------
[45/50] incubator-beam git commit: This closes #506
Posted by th...@apache.org.
This closes #506
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f6a9733f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f6a9733f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f6a9733f
Branch: refs/heads/apex-runner
Commit: f6a9733f58f6af5248608b7c433fee5a01e3b8b8
Parents: 70255d2 bc28799
Author: Jesse Anderson <je...@smokinghand.com>
Authored: Mon Nov 7 09:40:32 2016 -0800
Committer: Jesse Anderson <je...@smokinghand.com>
Committed: Mon Nov 7 09:40:32 2016 -0800
----------------------------------------------------------------------
.../beam/sdk/transforms/RegexTransform.java | 505 +++++++++++++++++++
.../beam/sdk/transforms/RegexTransformTest.java | 262 ++++++++++
2 files changed, 767 insertions(+)
----------------------------------------------------------------------
[28/50] incubator-beam git commit: This closes #1252
Posted by th...@apache.org.
This closes #1252
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/faf55c78
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/faf55c78
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/faf55c78
Branch: refs/heads/apex-runner
Commit: faf55c78a30a6937a8b598e674a48f446fcb5424
Parents: 3419fef 8bf6d92
Author: Thomas Groh <tg...@google.com>
Authored: Fri Nov 4 09:57:44 2016 -0700
Committer: Thomas Groh <tg...@google.com>
Committed: Fri Nov 4 09:57:44 2016 -0700
----------------------------------------------------------------------
.../sdk/transforms/reflect/DoFnInvokers.java | 7 +-
.../sdk/transforms/reflect/DoFnSignature.java | 44 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 752 ++++++++++++-------
.../apache/beam/sdk/values/TypeDescriptor.java | 13 +
.../DoFnSignaturesProcessElementTest.java | 18 +-
.../DoFnSignaturesSplittableDoFnTest.java | 19 +-
.../transforms/reflect/DoFnSignaturesTest.java | 42 +-
.../reflect/DoFnSignaturesTestUtils.java | 13 +-
8 files changed, 578 insertions(+), 330 deletions(-)
----------------------------------------------------------------------
[06/50] incubator-beam git commit: Upgrade
google-api-services-dataflow to v1b3-rev43-1.22.0
Posted by th...@apache.org.
Upgrade google-api-services-dataflow to v1b3-rev43-1.22.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8c2b99e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8c2b99e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8c2b99e9
Branch: refs/heads/apex-runner
Commit: 8c2b99e960edf3d9d6d6eda188c17c5729308edc
Parents: 4dec38c
Author: peihe <pe...@users.noreply.github.com>
Authored: Wed Nov 2 10:44:58 2016 -0700
Committer: Davor Bonaci <da...@google.com>
Committed: Wed Nov 2 16:50:59 2016 -0700
----------------------------------------------------------------------
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c2b99e9/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 51518ed..cb1e4d6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -106,7 +106,7 @@
<cloudresourcemanager.version>v1-rev6-1.22.0</cloudresourcemanager.version>
<pubsubgrpc.version>0.1.0</pubsubgrpc.version>
<clouddebugger.version>v2-rev8-1.22.0</clouddebugger.version>
- <dataflow.version>v1b3-rev42-1.22.0</dataflow.version>
+ <dataflow.version>v1b3-rev43-1.22.0</dataflow.version>
<dataflow.proto.version>0.5.160222</dataflow.proto.version>
<datastore.client.version>1.2.0</datastore.client.version>
<datastore.proto.version>1.2.0</datastore.proto.version>
[09/50] incubator-beam git commit: This closes #1270
Posted by th...@apache.org.
This closes #1270
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6a05cf4a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6a05cf4a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6a05cf4a
Branch: refs/heads/apex-runner
Commit: 6a05cf4a94728eb5e73a96e260eb5ef0bdc5fc9b
Parents: d75d8b2 f233d85
Author: Sela <an...@paypal.com>
Authored: Thu Nov 3 15:35:10 2016 +0200
Committer: Sela <an...@paypal.com>
Committed: Thu Nov 3 15:35:10 2016 +0200
----------------------------------------------------------------------
runners/spark/pom.xml | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------