You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/02/28 17:44:59 UTC
[GitHub] [beam] apilloud opened a new pull request #11003: [BEAM-9411]
Enable BigQuery DIRECT_READ by default in SQL
apilloud opened a new pull request #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003
This is avalable globally now and we keep getting customer issues caused by it not being enabled. Lets turn it on by default!
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592731427
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-596728268
Run SQL Postcommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592647170
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] 11moon11 commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
11moon11 commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-596182321
Looks good!
I think that it may be worth updating the documentation [1, 2] for BigQuery Table to make it clear that DIRECT_READ is the default read method. But it can be done in a separate PR.
[1] https://beam.apache.org/documentation/dsls/sql/extensions/create-external-table/
[2] https://github.com/apache/beam/blob/master/website/src/documentation/dsls/sql/extensions/create-external-table.md
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592661992
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592681527
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592693089
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592673405
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kennknowles commented on a change in pull request #11003:
[BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#discussion_r389893825
##########
File path: sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java
##########
@@ -69,9 +73,18 @@
@Test
public void testRead() throws Exception {
- bigQuery.insertRows(ID_NAME_SCHEMA, row(1, "name1"), row(2, "name2"), row(3, "name3"));
-
TableReference bqTable = bigQuery.tableReference();
+
+ // Streaming inserts do not work with DIRECT_READ mode, there is a several hour lag.
+ PCollection<Row> data =
+ writePipeline.apply(Create.of(row(1, "name1"), row(2, "name2"), row(3, "name3")));
+ data.apply(
+ BigQueryIO.<Row>write()
+ .withSchema(BigQueryUtils.toTableSchema(ID_NAME_SCHEMA))
+ .withFormatFunction(BigQueryUtils.toTableRow())
Review comment:
I would suggest to add `.withMethod(Method.FILE_LOADS)` to be explicit. It connects it better with the comment above. And is protected in case the default changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on a change in pull request #11003:
[BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on a change in pull request #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#discussion_r389903110
##########
File path: sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java
##########
@@ -69,9 +73,18 @@
@Test
public void testRead() throws Exception {
- bigQuery.insertRows(ID_NAME_SCHEMA, row(1, "name1"), row(2, "name2"), row(3, "name3"));
-
TableReference bqTable = bigQuery.tableReference();
+
+ // Streaming inserts do not work with DIRECT_READ mode, there is a several hour lag.
+ PCollection<Row> data =
+ writePipeline.apply(Create.of(row(1, "name1"), row(2, "name2"), row(3, "name3")));
+ data.apply(
+ BigQueryIO.<Row>write()
+ .withSchema(BigQueryUtils.toTableSchema(ID_NAME_SCHEMA))
+ .withFormatFunction(BigQueryUtils.toTableRow())
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592692026
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-595917427
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud merged pull request #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud merged pull request #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592735040
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-595931935
R: @kennknowles Please take another look.
cc: @11moon11
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-595904300
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-596702036
Good point, updated the website as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592681235
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] apilloud commented on issue #11003: [BEAM-9411] Enable
BigQuery DIRECT_READ by default in SQL
Posted by GitBox <gi...@apache.org>.
apilloud commented on issue #11003: [BEAM-9411] Enable BigQuery DIRECT_READ by default in SQL
URL: https://github.com/apache/beam/pull/11003#issuecomment-592674707
Yes, both paths have test coverage. Will merge as soon as I figure out the timing issues in the read test. (Test data isn't being committed to BigQuery before we try to read it, this change makes things faster which causes problems.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services