You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/03 16:54:40 UTC
[GitHub] [beam] kmjung opened a new pull request #12767: Remove experimental annotations for BQ storage API source
kmjung opened a new pull request #12767:
URL: https://github.com/apache/beam/pull/12767
This change removes the Experimental annotations for BigQuery storage API sources. The storage API is now GA, and the sources have been updated to use the GA API surface.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kmjung commented on a change in pull request #12767: [BEAM-10916] Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
kmjung commented on a change in pull request #12767:
URL: https://github.com/apache/beam/pull/12767#discussion_r489970520
##########
File path: website/www/site/content/en/documentation/io/built-in/google-bigquery.md
##########
@@ -322,21 +322,17 @@ in the following example:
### Using the BigQuery Storage API {#storage-api}
The [BigQuery Storage API](https://cloud.google.com/bigquery/docs/reference/storage/)
-allows you to directly access tables in BigQuery storage. As a result, your
-pipeline can read from BigQuery storage faster than previously possible.
+allows you to directly access tables in BigQuery storage, and supports features
+such as column selection and predicate filter push-down which can allow more
+efficient pipeline execution.
-The Beam SDK for Java (version 2.11.0 and later) adds support for the beta
-release of the BigQuery Storage API as an [experimental feature](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html).
-Beam's support for the BigQuery Storage API has the following limitations:
+The Beam SDK for Java supports using the BigQuery Storage API when reading from
+BigQuery. SDK versions before 2.24.0 support the BigQuery Storage API as an
+[experimental feature](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html)
+and use the pre-GA BigQuery Storage API surface. Callers should migrate
+pipelines which use the BigQuery Storage API to use SDK version 2.24.0 or later.
-* The SDK for Python does not support the BigQuery Storage API.
-* SDK versions 2.11.0 and 2.12.0 do not support reading with a query string; you
- can only read from a table.
-* SDK versions before 2.15.0 do not support dynamic work rebalancing. As a
- result, reads might be less efficient in the presence of stragglers.
-
-Because this is currently a Beam experimental feature, export based reads are
-recommended for production jobs.
+The Beam SDK for Python does not support the BigQuery Storage API.
Review comment:
Sorry, to be clear, do you mean a JIRA feature request to support Python?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #12767: Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #12767:
URL: https://github.com/apache/beam/pull/12767#discussion_r485996726
##########
File path: website/www/site/content/en/documentation/io/built-in/google-bigquery.md
##########
@@ -322,21 +322,17 @@ in the following example:
### Using the BigQuery Storage API {#storage-api}
The [BigQuery Storage API](https://cloud.google.com/bigquery/docs/reference/storage/)
-allows you to directly access tables in BigQuery storage. As a result, your
-pipeline can read from BigQuery storage faster than previously possible.
+allows you to directly access tables in BigQuery storage, and supports features
+such as column selection and predicate filter push-down which can allow more
+efficient pipeline execution.
-The Beam SDK for Java (version 2.11.0 and later) adds support for the beta
-release of the BigQuery Storage API as an [experimental feature](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html).
-Beam's support for the BigQuery Storage API has the following limitations:
+The Beam SDK for Java supports using the BigQuery Storage API when reading from
+BigQuery. SDK versions before 2.24.0 support the BigQuery Storage API as an
+[experimental feature](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html)
+and use the pre-GA BigQuery Storage API surface. Callers should migrate
+pipelines which use the BigQuery Storage API to use SDK version 2.24.0 or later.
-* The SDK for Python does not support the BigQuery Storage API.
-* SDK versions 2.11.0 and 2.12.0 do not support reading with a query string; you
- can only read from a table.
-* SDK versions before 2.15.0 do not support dynamic work rebalancing. As a
- result, reads might be less efficient in the presence of stragglers.
-
-Because this is currently a Beam experimental feature, export based reads are
-recommended for production jobs.
+The Beam SDK for Python does not support the BigQuery Storage API.
Review comment:
Can you create a JIRA and link it here ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kmjung commented on a change in pull request #12767: [BEAM-10916] Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
kmjung commented on a change in pull request #12767:
URL: https://github.com/apache/beam/pull/12767#discussion_r489989080
##########
File path: website/www/site/content/en/documentation/io/built-in/google-bigquery.md
##########
@@ -322,21 +322,17 @@ in the following example:
### Using the BigQuery Storage API {#storage-api}
The [BigQuery Storage API](https://cloud.google.com/bigquery/docs/reference/storage/)
-allows you to directly access tables in BigQuery storage. As a result, your
-pipeline can read from BigQuery storage faster than previously possible.
+allows you to directly access tables in BigQuery storage, and supports features
+such as column selection and predicate filter push-down which can allow more
+efficient pipeline execution.
-The Beam SDK for Java (version 2.11.0 and later) adds support for the beta
-release of the BigQuery Storage API as an [experimental feature](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html).
-Beam's support for the BigQuery Storage API has the following limitations:
+The Beam SDK for Java supports using the BigQuery Storage API when reading from
+BigQuery. SDK versions before 2.24.0 support the BigQuery Storage API as an
+[experimental feature](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html)
+and use the pre-GA BigQuery Storage API surface. Callers should migrate
+pipelines which use the BigQuery Storage API to use SDK version 2.24.0 or later.
-* The SDK for Python does not support the BigQuery Storage API.
-* SDK versions 2.11.0 and 2.12.0 do not support reading with a query string; you
- can only read from a table.
-* SDK versions before 2.15.0 do not support dynamic work rebalancing. As a
- result, reads might be less efficient in the presence of stragglers.
-
-Because this is currently a Beam experimental feature, export based reads are
-recommended for production jobs.
+The Beam SDK for Python does not support the BigQuery Storage API.
Review comment:
Opened [BEAM-10917](https://issues.apache.org/jira/browse/BEAM-10917) and added a link.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #12767: [BEAM-10916] Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #12767:
URL: https://github.com/apache/beam/pull/12767
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #12767: [BEAM-10916] Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #12767:
URL: https://github.com/apache/beam/pull/12767
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #12767: [BEAM-10916] Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #12767:
URL: https://github.com/apache/beam/pull/12767#discussion_r489971844
##########
File path: website/www/site/content/en/documentation/io/built-in/google-bigquery.md
##########
@@ -322,21 +322,17 @@ in the following example:
### Using the BigQuery Storage API {#storage-api}
The [BigQuery Storage API](https://cloud.google.com/bigquery/docs/reference/storage/)
-allows you to directly access tables in BigQuery storage. As a result, your
-pipeline can read from BigQuery storage faster than previously possible.
+allows you to directly access tables in BigQuery storage, and supports features
+such as column selection and predicate filter push-down which can allow more
+efficient pipeline execution.
-The Beam SDK for Java (version 2.11.0 and later) adds support for the beta
-release of the BigQuery Storage API as an [experimental feature](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html).
-Beam's support for the BigQuery Storage API has the following limitations:
+The Beam SDK for Java supports using the BigQuery Storage API when reading from
+BigQuery. SDK versions before 2.24.0 support the BigQuery Storage API as an
+[experimental feature](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html)
+and use the pre-GA BigQuery Storage API surface. Callers should migrate
+pipelines which use the BigQuery Storage API to use SDK version 2.24.0 or later.
-* The SDK for Python does not support the BigQuery Storage API.
-* SDK versions 2.11.0 and 2.12.0 do not support reading with a query string; you
- can only read from a table.
-* SDK versions before 2.15.0 do not support dynamic work rebalancing. As a
- result, reads might be less efficient in the presence of stragglers.
-
-Because this is currently a Beam experimental feature, export based reads are
-recommended for production jobs.
+The Beam SDK for Python does not support the BigQuery Storage API.
Review comment:
Yes. Feel free to add this or we can just merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kmjung commented on pull request #12767: Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
kmjung commented on pull request #12767:
URL: https://github.com/apache/beam/pull/12767#issuecomment-686624323
R: @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12767: Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12767:
URL: https://github.com/apache/beam/pull/12767#issuecomment-693010547
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12767: Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12767:
URL: https://github.com/apache/beam/pull/12767#issuecomment-693010464
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12767: [BEAM-10916] Remove experimental annotations for BQ storage API source
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12767:
URL: https://github.com/apache/beam/pull/12767#issuecomment-693856184
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org