You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/27 00:50:01 UTC
[GitHub] [beam] pabloem opened a new pull request #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
pabloem opened a new pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-610407671
LGTM. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r403350428
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -641,12 +641,19 @@ def __init__(
def estimate_size(self):
bq = bigquery_tools.BigQueryWrapper()
if self.table_reference is not None:
+ table_ref = self.table_reference
+ if (isinstance(self.table_reference, vp.ValueProvider) and
+ self.table_reference.is_accessible()):
+ table_ref = bigquery_tools.parse_table_reference(
+ self.table_reference.get(), self.dataset, self.project)
+ elif isinstance(self.table_reference, vp.ValueProvider):
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the table that we're querying.
+ return 0
table = bq.get_table(
- self.table_reference.projectId,
- self.table_reference.datasetId,
- self.table_reference.tableId)
+ table_ref.projectId, table_ref.datasetId, table_ref.tableId)
return int(table.numBytes)
- else:
+ elif self.query is not None and self.query.is_accessible():
Review comment:
It is supported now. In this case, the `self.query` parameter is wrapped by a ValueProvider already. This does not work for `self.table` because it may be a non-string.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r403350683
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -641,12 +641,19 @@ def __init__(
def estimate_size(self):
bq = bigquery_tools.BigQueryWrapper()
if self.table_reference is not None:
+ table_ref = self.table_reference
+ if (isinstance(self.table_reference, vp.ValueProvider) and
+ self.table_reference.is_accessible()):
+ table_ref = bigquery_tools.parse_table_reference(
+ self.table_reference.get(), self.dataset, self.project)
+ elif isinstance(self.table_reference, vp.ValueProvider):
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the table that we're querying.
+ return 0
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-604759269
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-608634124
i dont understand why it's failing on permission issues now..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-609723916
This time it failed because of `docker-credential-gcloud not installed or not available in PATH`. This is a known issue, we have a ticket for that: https://issues.apache.org/jira/browse/BEAM-8409
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-605510718
passing pc before formatting: https://builds.apache.org/job/beam_PostCommit_Python37_PR/122/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-606925872
r: @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r402669567
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -641,12 +641,19 @@ def __init__(
def estimate_size(self):
bq = bigquery_tools.BigQueryWrapper()
if self.table_reference is not None:
+ table_ref = self.table_reference
+ if (isinstance(self.table_reference, vp.ValueProvider) and
+ self.table_reference.is_accessible()):
+ table_ref = bigquery_tools.parse_table_reference(
+ self.table_reference.get(), self.dataset, self.project)
+ elif isinstance(self.table_reference, vp.ValueProvider):
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the table that we're querying.
+ return 0
Review comment:
We have to return None if size cannot be determined. See spec here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L157
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r402668065
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -641,12 +641,19 @@ def __init__(
def estimate_size(self):
bq = bigquery_tools.BigQueryWrapper()
if self.table_reference is not None:
+ table_ref = self.table_reference
+ if (isinstance(self.table_reference, vp.ValueProvider) and
+ self.table_reference.is_accessible()):
+ table_ref = bigquery_tools.parse_table_reference(
+ self.table_reference.get(), self.dataset, self.project)
+ elif isinstance(self.table_reference, vp.ValueProvider):
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the table that we're querying.
+ return 0
table = bq.get_table(
- self.table_reference.projectId,
- self.table_reference.datasetId,
- self.table_reference.tableId)
+ table_ref.projectId, table_ref.datasetId, table_ref.tableId)
return int(table.numBytes)
- else:
+ elif self.query is not None and self.query.is_accessible():
job = bq._start_query_job(
Review comment:
For from this PR but shouldn't this be a dry run ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r402669596
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -657,6 +664,10 @@ def estimate_size(self):
kms_key=self.kms_key)
size = int(job.statistics.totalBytesProcessed)
return size
+ else:
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the query that we're running.
+ return 0
Review comment:
Ditto.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-605181366
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-608768810
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r402668065
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -641,12 +641,19 @@ def __init__(
def estimate_size(self):
bq = bigquery_tools.BigQueryWrapper()
if self.table_reference is not None:
+ table_ref = self.table_reference
+ if (isinstance(self.table_reference, vp.ValueProvider) and
+ self.table_reference.is_accessible()):
+ table_ref = bigquery_tools.parse_table_reference(
+ self.table_reference.get(), self.dataset, self.project)
+ elif isinstance(self.table_reference, vp.ValueProvider):
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the table that we're querying.
+ return 0
table = bq.get_table(
- self.table_reference.projectId,
- self.table_reference.datasetId,
- self.table_reference.tableId)
+ table_ref.projectId, table_ref.datasetId, table_ref.tableId)
return int(table.numBytes)
- else:
+ elif self.query is not None and self.query.is_accessible():
job = bq._start_query_job(
Review comment:
Not from this PR but shouldn't this be a dry run ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem merged pull request #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-607159015
This looks great! Thank you @pabloem, I have nothing to add
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-610042807
@chamikaramj any other comments? I might merge as Kamil has approved and your ocmments have been addressed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-609723986
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r403350549
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -641,12 +641,19 @@ def __init__(
def estimate_size(self):
bq = bigquery_tools.BigQueryWrapper()
if self.table_reference is not None:
+ table_ref = self.table_reference
+ if (isinstance(self.table_reference, vp.ValueProvider) and
+ self.table_reference.is_accessible()):
+ table_ref = bigquery_tools.parse_table_reference(
+ self.table_reference.get(), self.dataset, self.project)
+ elif isinstance(self.table_reference, vp.ValueProvider):
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the table that we're querying.
+ return 0
table = bq.get_table(
- self.table_reference.projectId,
- self.table_reference.datasetId,
- self.table_reference.tableId)
+ table_ref.projectId, table_ref.datasetId, table_ref.tableId)
return int(table.numBytes)
- else:
+ elif self.query is not None and self.query.is_accessible():
job = bq._start_query_job(
Review comment:
if you look at line 663, it is a dry run : )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-610042507
Failure is HDFS integration test. unrelated to this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-607510701
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r402668608
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -641,12 +641,19 @@ def __init__(
def estimate_size(self):
bq = bigquery_tools.BigQueryWrapper()
if self.table_reference is not None:
+ table_ref = self.table_reference
+ if (isinstance(self.table_reference, vp.ValueProvider) and
+ self.table_reference.is_accessible()):
+ table_ref = bigquery_tools.parse_table_reference(
+ self.table_reference.get(), self.dataset, self.project)
+ elif isinstance(self.table_reference, vp.ValueProvider):
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the table that we're querying.
+ return 0
table = bq.get_table(
- self.table_reference.projectId,
- self.table_reference.datasetId,
- self.table_reference.tableId)
+ table_ref.projectId, table_ref.datasetId, table_ref.tableId)
return int(table.numBytes)
- else:
+ elif self.query is not None and self.query.is_accessible():
Review comment:
Shouldn't we support ValueProvider for queries as well ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11244: [BEAM-3097]
_ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#issuecomment-607421085
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11244:
[BEAM-3097] _ReadFromBigQuery supports valueprovider for table
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11244: [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
URL: https://github.com/apache/beam/pull/11244#discussion_r403350713
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -657,6 +664,10 @@ def estimate_size(self):
kms_key=self.kms_key)
size = int(job.statistics.totalBytesProcessed)
return size
+ else:
+ # Size estimation is best effort. We return 0 as we have no
+ # access to the query that we're running.
+ return 0
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services