You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/30 20:41:36 UTC
[GitHub] [beam] Hannah-Jiang opened a new pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Hannah-Jiang opened a new pull request #11584:
URL: https://github.com/apache/beam/pull/11584
R: @tvalentyn
Cc: @ibzib
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11584:
URL: https://github.com/apache/beam/pull/11584#issuecomment-628965737
@Hannah-Jiang - you can continue with this change now. However you will need to rebase.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Hannah-Jiang commented on a change in pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
Hannah-Jiang commented on a change in pull request #11584:
URL: https://github.com/apache/beam/pull/11584#discussion_r418278914
##########
File path: sdks/python/container/py2/build.gradle
##########
@@ -66,7 +66,8 @@ docker {
project.rootProject["docker-tag"] : project.sdk_version)
files "../Dockerfile", "./build"
buildArgs(['py_version': "2.7",
- 'pull_licenses': project.rootProject.hasProperty(["docker-pull-licenses"])])
+ 'pull_licenses': project.rootProject.hasProperty(["docker-pull-licenses"]) ||
+ project.rootProject.hasProperty(["isRelease"])])
Review comment:
Currently, the `isRelease` tag needs to be passed with commands. Ideally, this tag should be set by checking branch name.
Detecting `isRelease` for docker image build is not that helpful for now, but could be used when we support auto setting `isRelease` tag.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #11584:
URL: https://github.com/apache/beam/pull/11584#discussion_r418340633
##########
File path: sdks/python/container/py2/build.gradle
##########
@@ -66,7 +66,8 @@ docker {
project.rootProject["docker-tag"] : project.sdk_version)
files "../Dockerfile", "./build"
buildArgs(['py_version': "2.7",
- 'pull_licenses': project.rootProject.hasProperty(["docker-pull-licenses"])])
+ 'pull_licenses': project.rootProject.hasProperty(["docker-pull-licenses"]) ||
+ project.rootProject.hasProperty(["isRelease"])])
Review comment:
I see, thanks. I did not know that we need to set isRelease manually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Hannah-Jiang merged pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
Hannah-Jiang merged pull request #11584:
URL: https://github.com/apache/beam/pull/11584
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #11584:
URL: https://github.com/apache/beam/pull/11584#discussion_r418767547
##########
File path: website/src/contribute/release-guide.md
##########
@@ -688,8 +688,20 @@ Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
* Build Python images and push to DockerHub.
```
-./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+./gradlew :sdks:python:container:buildAll -Pdocker-pull-licenses -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+```
+
+Verify that third party licenses are included by logging in to the images. For Python SDK images, there should be around 80 ~ 100 dependencies.
+Please note that dependencies for the SDKs with different Python versions vary.
+Need to verify all Python images by replacing `${ver}` in the following command to `python2.7, python3.5, python3.6, python3.7`.
Review comment:
Ack, thanks for the heads up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11584:
URL: https://github.com/apache/beam/pull/11584#issuecomment-629369467
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Hannah-Jiang commented on pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
Hannah-Jiang commented on pull request #11584:
URL: https://github.com/apache/beam/pull/11584#issuecomment-629030902
It is rebased. Please take a look when you have time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Hannah-Jiang commented on a change in pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
Hannah-Jiang commented on a change in pull request #11584:
URL: https://github.com/apache/beam/pull/11584#discussion_r418278914
##########
File path: sdks/python/container/py2/build.gradle
##########
@@ -66,7 +66,8 @@ docker {
project.rootProject["docker-tag"] : project.sdk_version)
files "../Dockerfile", "./build"
buildArgs(['py_version': "2.7",
- 'pull_licenses': project.rootProject.hasProperty(["docker-pull-licenses"])])
+ 'pull_licenses': project.rootProject.hasProperty(["docker-pull-licenses"]) ||
+ project.rootProject.hasProperty(["isRelease"])])
Review comment:
Currently, the `isRelease` tag needs to be passed with commands. Ideally, this tag should be set by checking branch name.
Adding this support to docker images are not that helpful for now, but could be used when we support auto setting `isRelease` tag.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Hannah-Jiang commented on a change in pull request #11584: [BEAM-9136]support isRelease tag for docker build command & update release guide
Posted by GitBox <gi...@apache.org>.
Hannah-Jiang commented on a change in pull request #11584:
URL: https://github.com/apache/beam/pull/11584#discussion_r418766264
##########
File path: website/src/contribute/release-guide.md
##########
@@ -688,8 +688,20 @@ Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
* Build Python images and push to DockerHub.
```
-./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+./gradlew :sdks:python:container:buildAll -Pdocker-pull-licenses -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+```
+
+Verify that third party licenses are included by logging in to the images. For Python SDK images, there should be around 80 ~ 100 dependencies.
+Please note that dependencies for the SDKs with different Python versions vary.
+Need to verify all Python images by replacing `${ver}` in the following command to `python2.7, python3.5, python3.6, python3.7`.
Review comment:
@ibzib , this part should be verified BEFORE pushing Python images. I'm not sure when this PR can be merged because of website issues, so explicitly pinging you here.
##########
File path: website/src/contribute/release-guide.md
##########
@@ -699,7 +711,18 @@ done
* Build Java images and push to DockerHub.
```
-./gradlew :sdks:java:container:dockerPush -Pdocker-pull-licenses -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+./gradlew :sdks:java:container:docker -Pdocker-pull-licenses -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+```
+
+Verify that third party licenses are included by logging in to the images. For Java SDK images, there should be around 1400 dependencies.
+```
+docker run -it --entrypoint=/bin/bash apache/beam_java_sdk:${RELEASE}_rc{RC_NUM}
+ls -al /opt/apache/beam/third_party_licenses/ | wc -l
+```
+
+After verifying the third party licenses are included correctly, push the images to DockerHub.
+```
Review comment:
@ibzib , this part changed slightly. Please note the change from `dockerPush` to `docker` at L714.
After creating the Java image, we should verify it BEFORE pushing to DockerHub. I'm not sure when this PR can be merged because of website issues, so explicitly pinging you here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org