You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/04 10:23:29 UTC

[GitHub] [beam] EDjur opened a new pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

EDjur opened a new pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040
 
 
   This PR relates to https://issues.apache.org/jira/browse/BEAM-9305 and allows for the usage of ValueProvider query strings in _ReadFromBigQuery.
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
   XLang | --- | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website
   --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) 
   Portable | --- | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] pabloem commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r387850061
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery.py
 ##########
 @@ -644,6 +644,8 @@ def estimate_size(self):
           self.table_reference.tableId)
       return int(table.numBytes)
     else:
+      if isinstance(self.query, ValueProvider):
+        self.query = self.query.get()
 
 Review comment:
   I would probably do this without changing the class variable.
   
   I'd do a local variable: `query = self.query.get()`, and do it on every instance where `self.query` is accessed.
   
   The reason is that the Source objects may be recreated in different workers, and the  `self.query` variable would contain the original ValueProvider in all of those.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] TheNeuralBit commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-603537323
 
 
   Thank _you_ for catching it @pabloem :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-597869174
 
 
   I'm srry @EDjur - the `expand` method runs at template construction, so the change currently will not work with parameters provided at execution time. You'd need to call `get` on the valueprovider in estimate_size/split/etc like we had discussed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r392122769
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery.py
 ##########
 @@ -694,14 +699,16 @@ def read(self, range_tracker):
     raise NotImplementedError('BigQuery source must be split before being read')
 
   def _setup_temporary_dataset(self, bq):
+    query = self.evaluate_query_valueprovider()
 
 Review comment:
   Thanks for the tip re: `check_accessible`, I didn't know about that. I've updated the PR based on these suggestions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-599450124
 
 
   Sure, JIRA ticket is at https://issues.apache.org/jira/browse/BEAM-9506

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-597065772
 
 
   @kamilwu Looks like the `AnnotateText` PTransform mention in `CHANGES.md` was removed in https://github.com/apache/beam/commit/0cc577134b9e98b32020efab2ca07ccd1665f7cb#diff-8b1c3fd0d4a6765c16dfd18509182f9d.
   
   So I added it back again.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur edited a comment on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur edited a comment on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-594450634
 
 
   Since `estimate_size` is called before `split` and uses `self.query`, I realised the value provider evaluation actually needs to take place before `split` is called. Does this make sense @pabloem ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-602847544
 
 
   Thanks @EDjur for the contribution! Thanks @kamilwu for reviewing

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] pabloem merged pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-599459578
 
 
   LGTM, let's wait for @pabloem's final review as well

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-595154066
 
 
   How about something like this instead @pabloem ? I noticed that `gcs_location`supports ValueProviders and is wrapped & initialised in the `_ReadFromBigQuery` transform. This reduces clutter in the `_CustomBigQuerySource`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-597930882
 
 
   ah good catch. Yes, that would make it a bug..

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-594450634
 
 
   Since `estimate_size` is called before `split` and the uses `self.query`, I realised the value provider evaluation actually needs to take place before `split` is called. Does this make sense?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-595155257
 
 
   Adding @kamilwu as you were mentioned in the Jira ticket too.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] kamilwu commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r390250330
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
 ##########
 @@ -156,12 +157,19 @@ def test_native_source(self):
 
   @attr('IT')
   def test_iobase_source(self):
+    query = StaticValueProvider(str, self.query)
     with beam.Pipeline(argv=self.args) as p:
       result = (
           p | 'read' >> beam.io._ReadFromBigQuery(
               query=self.query, use_standard_sql=True, project=self.project))
       assert_that(result, equal_to(self.TABLE_DATA))
 
+    with beam.Pipeline(argv=self.args) as p:
 
 Review comment:
   @EDjur Thanks for explanation. That's right, if we want to check that query is successfully evaluated to a string, we have to execute the pipeline.
   @pabloem I think this is good idea. Let's do it this way.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] TheNeuralBit commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-603510136
 
 
   It looks like this change may have broken `apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python` in Python postcommits, e.g. https://builds.apache.org/job/beam_PostCommit_Python2/2036/
   
   Could you take a look @EDjur and/or @kamilwu?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-594440681
 
 
   R: @pabloem 
   
   As I have little experience with the BQ module, I'm a bit unsure if the test was placed in the correct module.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] kamilwu commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r391760791
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery.py
 ##########
 @@ -694,14 +699,16 @@ def read(self, range_tracker):
     raise NotImplementedError('BigQuery source must be split before being read')
 
   def _setup_temporary_dataset(self, bq):
+    query = self.evaluate_query_valueprovider()
 
 Review comment:
   There's a `check_accessible` decorator that could be used for this function (as well as for `estimate_size`).
   I think you could also test if `self.query` is a ValueProvider or not in the constructor. If it's not, you can then create a StaticValueProvider object. `self.evaluate_query_valueprovider()` would be unnecessary.
   
   Here'a en example:
   https://github.com/apache/beam/blob/f75838f306fc1d851b0dea11c2cb861b43430568/sdks/python/apache_beam/io/filebasedsource.py#L115-L117

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] pabloem commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r389835922
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
 ##########
 @@ -156,12 +157,19 @@ def test_native_source(self):
 
   @attr('IT')
   def test_iobase_source(self):
+    query = StaticValueProvider(str, self.query)
     with beam.Pipeline(argv=self.args) as p:
       result = (
           p | 'read' >> beam.io._ReadFromBigQuery(
               query=self.query, use_standard_sql=True, project=self.project))
       assert_that(result, equal_to(self.TABLE_DATA))
 
+    with beam.Pipeline(argv=self.args) as p:
 
 Review comment:
   What I would say is use the value provider in the first pipeline of this method, and don't add a second pipeline. I think that should be fine. WDYT @kamilwu ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-598308988
 
 
   >  Is that a bug then?
   
   Yes, this might be a bug. I don't know a good solution at the moment though... this is more complicated, because we have to fallback to temp_location from pipeline options if gcs_location is empty

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur edited a comment on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur edited a comment on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-594450634
 
 
   Since `estimate_size` is called before `split` and the uses `self.query`, I realised the value provider evaluation actually needs to take place before `split` is called. Does this make sense @pabloem ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-599442967
 
 
   @EDjur Could you create a JIRA ticket regarding `gcs_location` value provider evaluation and assign it to me afterwards? Thanks. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] pabloem commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r387847682
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
 ##########
 @@ -162,6 +163,15 @@ def test_iobase_source(self):
               query=self.query, use_standard_sql=True, project=self.project))
       assert_that(result, equal_to(self.TABLE_DATA))
 
+  @attr('IT')
+  def test_valueprovider_query_string(self):
+    query = StaticValueProvider(str, self.query)
+    with beam.Pipeline(argv=self.args) as p:
+      result = (
+          p | 'read' >> beam.io._ReadFromBigQuery(
+              query=query, use_standard_sql=True, project=self.project))
+      assert_that(result, equal_to(self.TABLE_DATA))
 
 Review comment:
   could you use the value provider in another one of the tests in this file? Since each test in this file takes ~5min to run, I'd prefer to avoid adding a new test.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-603511803
 
 
   Thanks for catching this @TheNeuralBit . I have https://github.com/apache/beam/pull/11206 out to fix this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-597872563
 
 
   The `gcs_location` value provider is evaluated in the `expand` method: https://github.com/apache/beam/pull/11040/files#diff-683ba04a0e750cf0456b9b119393ad64R1587. Is that a bug then?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r389532725
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
 ##########
 @@ -156,12 +157,19 @@ def test_native_source(self):
 
   @attr('IT')
   def test_iobase_source(self):
+    query = StaticValueProvider(str, self.query)
     with beam.Pipeline(argv=self.args) as p:
       result = (
           p | 'read' >> beam.io._ReadFromBigQuery(
               query=self.query, use_standard_sql=True, project=self.project))
       assert_that(result, equal_to(self.TABLE_DATA))
 
+    with beam.Pipeline(argv=self.args) as p:
 
 Review comment:
   To actually evaluate the value providers, the pipeline needs to be executed right? Cause these are realistically only 'known' at runtime. The tests in `TestReadFromBigQuery` just validate the input data to the initialisation.
   
   We can of course just validate that query is of type `ValueProvider` after initialising a pipeline, but I figured we want to check that the value provider query is actually evaluated to a string type on pipeline execution.
   
   Am I perhaps missing something?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] EDjur edited a comment on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
EDjur edited a comment on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#issuecomment-597872563
 
 
   The `gcs_location` value provider is evaluated in the `expand` method: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L1575. Is that a bug then?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [beam] kamilwu commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource

Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r388788211
 
 

 ##########
 File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
 ##########
 @@ -156,12 +157,19 @@ def test_native_source(self):
 
   @attr('IT')
   def test_iobase_source(self):
+    query = StaticValueProvider(str, self.query)
     with beam.Pipeline(argv=self.args) as p:
       result = (
           p | 'read' >> beam.io._ReadFromBigQuery(
               query=self.query, use_standard_sql=True, project=self.project))
       assert_that(result, equal_to(self.TABLE_DATA))
 
+    with beam.Pipeline(argv=self.args) as p:
 
 Review comment:
   Do we need an integration test to cover your value provider's logic? Take a look at `TestReadFromBigQuery` test case in `bigquery_test.py`. Those are unit tests for `_ReadFromBigQuery` that actually run very fast, because pipeline is not executed.
   
   Adding new tests in `bigquery_read_it_test.py` causes huge overhead, since they are all executed by three runners (Direct, Dataflow and Flink) on every supported Python version.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services