You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ch...@apache.org on 2017/09/25 18:11:01 UTC

[1/2] beam git commit: Updates WriteToBigQuery PTransform to get project id from GoogleCloudOptions when using DirectRunner.

Repository: beam
Updated Branches:
  refs/heads/master 06809b982 -> a2ff162c9


Updates WriteToBigQuery PTransform to get project id from GoogleCloudOptions when using DirectRunner.

WriteToBigQuery PTransform behaves differently for DirectRunner and DataflowRunner when it comes to determining the project that the output table belongs to. If a project is not specified, DataflowRunner defauls to GoogleCloudOptions.project while DirectRunner does not. This PR fixes this inconsistency by defaulting to GoogleCloudOptions.project for DirectRunner as well.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ae810034
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ae810034
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ae810034

Branch: refs/heads/master
Commit: ae8100348f43e50d27a500d4196fa08137ad81b8
Parents: 06809b9
Author: chamikara@google.com <ch...@google.com>
Authored: Fri Sep 22 17:59:50 2017 -0700
Committer: chamikara@google.com <ch...@google.com>
Committed: Mon Sep 25 11:10:22 2017 -0700

----------------------------------------------------------------------
 sdks/python/apache_beam/io/gcp/bigquery.py | 3 +++
 1 file changed, 3 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/beam/blob/ae810034/sdks/python/apache_beam/io/gcp/bigquery.py
----------------------------------------------------------------------
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py b/sdks/python/apache_beam/io/gcp/bigquery.py
index 40611ac..ee79ae5 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -1412,6 +1412,9 @@ bigquery_v2_messages.TableSchema):
       raise TypeError('Unexpected schema argument: %s.' % schema)
 
   def expand(self, pcoll):
+    if self.table_reference.projectId is None:
+      self.table_reference.projectId = pcoll.pipeline.options.view_as(
+          GoogleCloudOptions).project
     bigquery_write_fn = BigQueryWriteFn(
         table_id=self.table_reference.tableId,
         dataset_id=self.table_reference.datasetId,


[2/2] beam git commit: This closes #3892

Posted by ch...@apache.org.
This closes #3892


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a2ff162c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a2ff162c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a2ff162c

Branch: refs/heads/master
Commit: a2ff162c93e1cb812412b2b71f3d13a961c1d739
Parents: 06809b9 ae81003
Author: chamikara@google.com <ch...@google.com>
Authored: Mon Sep 25 11:10:44 2017 -0700
Committer: chamikara@google.com <ch...@google.com>
Committed: Mon Sep 25 11:10:44 2017 -0700

----------------------------------------------------------------------
 sdks/python/apache_beam/io/gcp/bigquery.py | 3 +++
 1 file changed, 3 insertions(+)
----------------------------------------------------------------------