You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/27 17:40:24 UTC
[GitHub] [beam] aaltay opened a new pull request #11251: Merge pull request
#11245 from apache/aaltay-patch-1
aaltay opened a new pull request #11251: Merge pull request #11245 from apache/aaltay-patch-1
URL: https://github.com/apache/beam/pull/11251
Update the range for pyarrow to support new pyarrow version 0.16.0
> Is this a regression from a previous release?
Kind of. Latest version of pyarrow is not supported blocking a down stream users next release (TFX).
> Is this a new feature or related to a new feature?
No.
> Would this impact production workloads for users?
Maybe. Depends on what is pre-installed in containers. We do not need to change what is pre-installed and most users would not have an issue.
> What percentage of users would be impacted by this issue if it is not fixed?
TFX users. (Probably >10% user base.)
> Would it be possible for the impacted users to skip this version?
Yes, but they will need to delay their release until 2.21.0.
@amaliujia - Please review this. I am not sure if this is a clear cut case for a cherry pick, I will leave up to you to decide.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605323799
@aaltay I will take care of this PR to get it merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605330242
> @aaltay I will take care of this PR to get it merged.
Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] brills edited a comment on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
brills edited a comment on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605152122
Thanks @aaltay -
To add some context and to clarify a bit on the severity of the issue:
pyarrow 0.16 contains new features and bug fixes that we (TFX) care about (just to list a few: https://github.com/apache/arrow/pull/6066
https://github.com/apache/arrow/pull/6006). So it would be nice if TFX can start using it in the next version (we were planning to release it at the end of April). But by that time, the latest beam would be 2.20.
Note that this is not to say if this PR is not merged, any bad consequence will happen, because we can also decide to not start using pyarrow 0.16, and still rely on / add workarounds on our end. it's just more technical debt.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605522390
I don't think the failed `python precommit` is related to this change. Will merge this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605400638
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605328759
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia merged pull request #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
amaliujia merged pull request #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605360886
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay removed a comment on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay removed a comment on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605295380
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605201122
Python pre commit failed with an unrelated error:
11:24:10 > Failed to list versions for com.google.errorprone:error_prone_core.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605201208
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605295535
I believe last test failed with another flake.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605295207
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay removed a comment on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay removed a comment on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605295207
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] brills commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
brills commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605152122
Thanks @aaltay -
To add some context:
pyarrow 0.16 contains new features and bug fixes that we (TFX) care about (just to list a few: https://github.com/apache/arrow/pull/6066
https://github.com/apache/arrow/pull/6006). So it would be nice it TFX can start using it in the next version (we were planning to release it at the end of April). But by that time, the latest beam would be 2.20.
Note that this is not to say if this PR is not merged, any bad consequence will happen, because we can also decide to not start using pyarrow 0.16, and still rely on / add workarounds on our end. it's just more technical debt.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605295380
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605392245
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605295558
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] brills edited a comment on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
brills edited a comment on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605152122
Thanks @aaltay -
To add some context:
pyarrow 0.16 contains new features and bug fixes that we (TFX) care about (just to list a few: https://github.com/apache/arrow/pull/6066
https://github.com/apache/arrow/pull/6006). So it would be nice if TFX can start using it in the next version (we were planning to release it at the end of April). But by that time, the latest beam would be 2.20.
Note that this is not to say if this PR is not merged, any bad consequence will happen, because we can also decide to not start using pyarrow 0.16, and still rely on / add workarounds on our end. it's just more technical debt.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605296166
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11251: Merge pull request #11245
- Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-606099473
> I don't think the failed `python precommit` is related to this change. Will merge this PR.
Thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11251: Merge pull request
#11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #11251: Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0
URL: https://github.com/apache/beam/pull/11251#issuecomment-605205313
LGTM. Will merge this PR after tests finish.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services