You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/28 19:19:25 UTC
[GitHub] [beam] pabloem opened a new pull request #11560: Auto-inferring project for ReadFromBigQuery
pabloem opened a new pull request #11560:
URL: https://github.com/apache/beam/pull/11560
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-622670110
PAssing precommit: https://builds.apache.org/job/beam_PostCommit_Python37_PR/132/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #11560:
URL: https://github.com/apache/beam/pull/11560#discussion_r419751064
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -526,7 +526,7 @@ def reader(self, test_bigquery_client=None):
def _to_bool(value):
- return value == 'true'
+ return value
Review comment:
could you explain why this change is needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-620888999
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-622986753
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-622999230
Python2_PVR_Flink tests are broken on master : /
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a change in pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
y1chi commented on a change in pull request #11560:
URL: https://github.com/apache/beam/pull/11560#discussion_r419804731
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -526,7 +526,7 @@ def reader(self, test_bigquery_client=None):
def _to_bool(value):
- return value == 'true'
+ return value
Review comment:
Do we need the _to_bool function then, would the bool casting suffice?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-623677009
@kamilwu @y1chi PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-620985048
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-622669830
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-622986688
Run Python Precommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-622993069
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11560:
URL: https://github.com/apache/beam/pull/11560#discussion_r419805319
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -526,7 +526,7 @@ def reader(self, test_bigquery_client=None):
def _to_bool(value):
- return value == 'true'
+ return value
Review comment:
ahhh what a great catch. thanks Yichi!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-620986934
py37 postcommit: https://builds.apache.org/job/beam_PostCommit_Python37_PR/128/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-621374775
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-623779892
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-621395376
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-620928773
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-623669571
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11560:
URL: https://github.com/apache/beam/pull/11560#discussion_r419765325
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -526,7 +526,7 @@ def reader(self, test_bigquery_client=None):
def _to_bool(value):
- return value == 'true'
+ return value
Review comment:
the coder incorrectly expected boolean types to be encoded as strings. This is incorrect, as JSON supports boolean 'natively', like this: https://json-schema.org/understanding-json-schema/reference/boolean.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11560: Auto-inferring project for ReadFromBigQuery
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11560:
URL: https://github.com/apache/beam/pull/11560#issuecomment-622611048
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org