You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/05/18 14:31:03 UTC
[GitHub] [beam] kamilwu opened a new pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
kamilwu opened a new pull request #11738:
URL: https://github.com/apache/beam/pull/11738
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-630222904
Run Python Dataflow ValidatesContainer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-631374566
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on a change in pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11738:
URL: https://github.com/apache/beam/pull/11738#discussion_r427859015
##########
File path: sdks/python/container/run_validatescontainer.sh
##########
@@ -69,8 +71,13 @@ elif [[ $1 == "python37" ]]; then
CONTAINER_PROJECT="sdks:python:container:py37" # Use this to build container by Gradle.
GRADLE_PY3_FLAG="-Ppython3" # Use this in Gradle command.
PY_INTERPRETER="python3.7" # Use this in virtualenv command.
+elif [[ $1 == "python38" ]]; then
+ IMAGE_NAME="${IMAGE_PREFIX}python3.8_sdk" # Use this to create CONTAINER_IMAGE variable.
+ CONTAINER_PROJECT="sdks:python:container:py38" # Use this to build container by Gradle.
+ GRADLE_PY3_FLAG="-Ppython3" # Use this in Gradle command.
Review comment:
Indeed there's no difference after removal. Looks like it was a leftover junk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-631629202
Run Python Dataflow ValidatesContainer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-631634279
> @ibzib Do you know if we will need to update release instructions to make sure we build and release Python 3.8 containers for future releases? (It may be done in a future PR).
All we need is to add 3.8 to these two scripts:
https://github.com/apache/beam/blob/836d0ad5b190502ac2086b9c44f4a7e8f4f9625d/release/src/main/scripts/build_release_candidate.sh#L51
https://github.com/apache/beam/blob/836d0ad5b190502ac2086b9c44f4a7e8f4f9625d/release/src/main/scripts/publish_docker_images.sh#L30
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-631628792
@ibzib Do you know if we will need to update release instructions to make sure we build and release Python 3.8 containers for future releases? (It may be done in a future PR).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-634552389
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-630752705
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-634781779
LGTM, thanks, @kamilwu !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn merged pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
tvalentyn merged pull request #11738:
URL: https://github.com/apache/beam/pull/11738
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-634557941
Run SQL_Java11 PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-631640224
Awesome, thank you!
I also opened BEAM-10046 to make sure Python 3.8 is reflected in the Beam release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-649032247
I think this PR might have caused BEAM-10316, PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-630739794
Run Python Dataflow ValidatesContainer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-634546732
Run Python Dataflow ValidatesContainer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-630226429
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-631398475
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-630920056
R: @Hannah-Jiang
cc: @tvalentyn
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
ibzib commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-631639325
> > @ibzib Do you know if we will need to update release instructions to make sure we build and release Python 3.8 containers for future releases? (It may be done in a future PR).
>
> All we need is to add 3.8 to these two scripts:
> https://github.com/apache/beam/blob/836d0ad5b190502ac2086b9c44f4a7e8f4f9625d/release/src/main/scripts/build_release_candidate.sh#L51
>
>
> https://github.com/apache/beam/blob/836d0ad5b190502ac2086b9c44f4a7e8f4f9625d/release/src/main/scripts/publish_docker_images.sh#L30
I also filed a ticket with INFRA to add the new repo on Docker hub, since no one on the Beam project has those privileges: https://issues.apache.org/jira/browse/INFRA-20288
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #11738:
URL: https://github.com/apache/beam/pull/11738#discussion_r427466990
##########
File path: .test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##########
@@ -39,5 +39,7 @@ PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
shell('cd ' + commonJobProperties.checkoutDir + ' && bash sdks/python/container/run_validatescontainer.sh python35')
shell('cd ' + commonJobProperties.checkoutDir + ' && bash sdks/python/container/run_validatescontainer.sh python36')
shell('cd ' + commonJobProperties.checkoutDir + ' && bash sdks/python/container/run_validatescontainer.sh python37')
+ // TODO: Turn on ValidatesContainer tests on Python 3.8 once BEAM-9754 is resolved.
Review comment:
Let's mention it as: TODO(BEAM-9754): ...
##########
File path: sdks/python/container/run_validatescontainer.sh
##########
@@ -69,8 +71,13 @@ elif [[ $1 == "python37" ]]; then
CONTAINER_PROJECT="sdks:python:container:py37" # Use this to build container by Gradle.
GRADLE_PY3_FLAG="-Ppython3" # Use this in Gradle command.
PY_INTERPRETER="python3.7" # Use this in virtualenv command.
+elif [[ $1 == "python38" ]]; then
+ IMAGE_NAME="${IMAGE_PREFIX}python3.8_sdk" # Use this to create CONTAINER_IMAGE variable.
+ CONTAINER_PROJECT="sdks:python:container:py38" # Use this to build container by Gradle.
+ GRADLE_PY3_FLAG="-Ppython3" # Use this in Gradle command.
Review comment:
` GRADLE_PY3_FLAG="-Ppython3" # Use this in Gradle command.` - is this still required? I don't quite understand what it does... Do you see any difference if we remove it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11738: [BEAM-9936] Create SDK harness containers with Python 3.8
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11738:
URL: https://github.com/apache/beam/pull/11738#issuecomment-630835922
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org