You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/12 00:19:04 UTC
[GitHub] [beam] pabloem opened a new pull request #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem opened a new pull request #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106
Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT.test_file_loads which are causing trouble at head.
The problem here is that AVRO does not encode b'1' into int(1) thus failing the pipeline. I would say this test relies on the BQ implementation detail that is able to import {'number': '1'} json as numeric. This fixes the test. Another possible fix of the test is to parse the bytes into actual numbers.
r: @chamikaramj
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598017978
sample py37 postcom: https://builds.apache.org/job/beam_PostCommit_Python37_PR/104/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#discussion_r392272672
##########
File path: sdks/python/apache_beam/options/pipeline_options.py
##########
@@ -1048,7 +1048,7 @@ def _add_argparse_args(cls, parser):
class FlinkRunnerOptions(PipelineOptions):
- PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9']
+ PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9', '1.10']
Review comment:
It shouldn't be too hard to have a central file with a list and read that list from these languages. We would have to have some form of build macro here though, to generate a Python file with the versions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-597980134
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ibzib commented on a change in pull request #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#discussion_r394550642
##########
File path: sdks/python/apache_beam/options/pipeline_options.py
##########
@@ -1048,7 +1048,7 @@ def _add_argparse_args(cls, parser):
class FlinkRunnerOptions(PipelineOptions):
- PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9']
+ PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9', '1.10']
Review comment:
Sounds good. I filed BEAM-9541 for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ibzib commented on a change in pull request #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#discussion_r391953302
##########
File path: sdks/python/apache_beam/options/pipeline_options.py
##########
@@ -1048,7 +1048,7 @@ def _add_argparse_args(cls, parser):
class FlinkRunnerOptions(PipelineOptions):
- PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9']
+ PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9', '1.10']
Review comment:
Unfortunately, such a document would itself need to stay up-to-date.
It'd be great if we could have a single source of truth, but it might be somewhat challenging considering the number of different places we need it (Python, Bash, Gradle, etc.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] Hannah-Jiang commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
Hannah-Jiang commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598029301
> fyi @Hannah-Jiang I'm trying to fix python postcommits in this PR : )
Thanks for letting me know. Which PR introduced the failure?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598022746
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598029192
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#discussion_r391751893
##########
File path: sdks/python/apache_beam/options/pipeline_options.py
##########
@@ -1048,7 +1048,7 @@ def _add_argparse_args(cls, parser):
class FlinkRunnerOptions(PipelineOptions):
- PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9']
+ PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9', '1.10']
Review comment:
Unfortunately, we don't. I've suggested to infer this information dynamically from the build but that also caused problems in the past; that's why we reverted to the manual handling. We should document all places which need adjustment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598026769
LGTM. Thanks.
Should we document differences in semantics in BQ sink when using Avro and JSON formats somewhere ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598029240
Run Python 3.5 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-597944001
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-597984563
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598029216
Run Python 3.6 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-597984578
Run Python 2 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598026604
fyi @Hannah-Jiang I'm trying to fix python postcommits in this PR : )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-597943970
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on a change in pull request #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#discussion_r391745372
##########
File path: sdks/python/apache_beam/options/pipeline_options.py
##########
@@ -1048,7 +1048,7 @@ def _add_argparse_args(cls, parser):
class FlinkRunnerOptions(PipelineOptions):
- PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9']
+ PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9', '1.10']
Review comment:
Do we have a process to keep this list up to date?
@mxm @ibzib
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106#issuecomment-598017085
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay merged pull request #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Posted by GitBox <gi...@apache.org>.
aaltay merged pull request #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services