You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2017/05/02 22:33:48 UTC
[1/3] beam git commit: Instantiate runner briefly in Pipeline
Repository: beam
Updated Branches:
refs/heads/master a552fb8f6 -> 027dd777d
Instantiate runner briefly in Pipeline
Today, some runners mutate the PipelineOptions in critical ways
when they are built, and BigQueryIO depends, during construction,
on options that are sometimes the subject of those mutations.
Changes to BigQueryIO and runners will likely fix this, but this
commit unbreaks a postcommit build.
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/78b25723
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/78b25723
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/78b25723
Branch: refs/heads/master
Commit: 78b257237e3fa02d868c8a3a0ed898de4c59fd0b
Parents: 6d9b239
Author: Kenneth Knowles <kl...@google.com>
Authored: Mon May 1 09:57:06 2017 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Tue May 2 14:01:15 2017 -0700
----------------------------------------------------------------------
sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam/blob/78b25723/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
index d4c46cc..ab8906a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
@@ -148,6 +148,9 @@ public class Pipeline {
* @return The newly created pipeline.
*/
public static Pipeline create(PipelineOptions options) {
+ // TODO: fix runners that mutate PipelineOptions in this method, then remove this line
+ PipelineRunner.fromOptions(options);
+
Pipeline pipeline = new Pipeline(options);
LOG.debug("Creating {}", pipeline);
return pipeline;
[3/3] beam git commit: This closes #2795: Restore status quo
relationship between PipelineOptions initialization and PipelineRunners
Posted by ke...@apache.org.
This closes #2795: Restore status quo relationship between PipelineOptions initialization and PipelineRunners
Instantiate runner briefly in Pipeline
Skip null options when converting back to argv
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/027dd777
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/027dd777
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/027dd777
Branch: refs/heads/master
Commit: 027dd777dd3c207cc689b4e1b350ca984d47191a
Parents: a552fb8 78b2572
Author: Kenneth Knowles <kl...@google.com>
Authored: Tue May 2 15:33:28 2017 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Tue May 2 15:33:28 2017 -0700
----------------------------------------------------------------------
sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java | 3 +++
.../src/main/java/org/apache/beam/sdk/testing/TestPipeline.java | 4 +++-
2 files changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
[2/3] beam git commit: Skip null options when converting back to argv
Posted by ke...@apache.org.
Skip null options when converting back to argv
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6d9b2393
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6d9b2393
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6d9b2393
Branch: refs/heads/master
Commit: 6d9b239350bb43a7bbe8b0c6ba1b20b806e2a03e
Parents: 4d6f6a1
Author: Kenneth Knowles <kl...@google.com>
Authored: Mon May 1 10:25:37 2017 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Tue May 2 14:01:15 2017 -0700
----------------------------------------------------------------------
.../src/main/java/org/apache/beam/sdk/testing/TestPipeline.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam/blob/6d9b2393/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
----------------------------------------------------------------------
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
index d45106c..ab8772b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
@@ -422,7 +422,9 @@ public class TestPipeline extends Pipeline implements TestRule {
Iterator<Entry<String, JsonNode>> entries = optsNode.fields();
while (entries.hasNext()) {
Entry<String, JsonNode> entry = entries.next();
- if (entry.getValue().isTextual()) {
+ if (entry.getValue().isNull()) {
+ continue;
+ } else if (entry.getValue().isTextual()) {
optArrayList.add("--" + entry.getKey() + "=" + entry.getValue().asText());
} else {
optArrayList.add("--" + entry.getKey() + "=" + entry.getValue());