You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/09/28 18:22:37 UTC
[GitHub] [incubator-mxnet] waytrue17 opened a new pull request #19244: [WIP] Website release pipeline automation
waytrue17 opened a new pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244
All versions of MXNet website artifacts are compressed into a zip file and stored in a private S3 bucket, with the following file structure:
> version
> ----1.0.0
> ----1.1.0
>...
> ----1.6.0
> ----1.7.0
Currently, for new MXNet versions to release, we have a pipeline that generates version specific artifacts, but it's still a manual process to upload them to s3 in the expected format shown above.
This PR aims to automate the website release pipeline.
We introduced pipeline `Jenkinsfile_website_release` to build the new version website artifacts, prepare all versions of website artifacts in to the expected format, and upload them to the target S3 bucket. This pipeline utilizes Jenkins global variable `RELEASE_WEBSITE_VERSION` to specify the name of the new version to be generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
mseth10 commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-706428590
Yes, this needs to be backported to 1.x and master. @waytrue17 can you create those PRs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r499036656
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,61 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
Review comment:
@marcoabreu removed archiving call, please review. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] ys2843 commented on a change in pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
ys2843 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497242109
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,27 @@ build_docs_beta() {
popd
}
+push_docs() {
+ folder_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $folder_name && tar -xzf full_website.tgz -C $folder_name --strip-components 1
+ # check if folder_name already exists in versions
+ pushd versions
+ if [ -d "$folder_name" ]; then
+ echo "Folder $folder_name already exists in versions. Please double check the FOLDER_NAME variable in Jenkens pipeline"
+ exit 1
+ fi
+ popd
+ mv $folder_name versions
+ zip -r9 versions.zip versions/.
+ aws s3 cp versions.zip s3://mxnet-website-static-artifacts --acl bucket-owner-full-control
Review comment:
Is the new artifact uploaded to S3 bucket accessible by the pipeline?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-700498194
The PR changes are being tested by this Jenkins pipeline: http://jenkins.mxnet-ci-dev.amazon-ml.com/job/docs/job/sethman-website-artifacts/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r499036821
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,61 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
+ }
+ }
+ }
+ }]
+}
+
+// This is for uploading website artifacts to S3 bucket
+// Assumes you have run docs_full_website function
+def docs_upload_s3() {
+ return ['Upload artifacts to s3 bucket': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ if(env.FOLDER_NAME) {
+ utils.init_git()
+
+ unstash 'full_website'
+
Review comment:
switched to `utils.unpack_and_init` and tested it on prod account
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-702482866
Jenkins CI successfully triggered : [unix-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497788184
##########
File path: ci/jenkins/Jenkinsfile_website_release
##########
@@ -0,0 +1,67 @@
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// timeout in minutes
+max_time = 180
+
+node('utility') {
+ // Loading the utilities requires a node context unfortunately
+ checkout scm
+ utils = load('ci/Jenkinsfile_utils.groovy')
+ custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
+}
+
+utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')
Review comment:
@marcoabreu Just to clarify, this pipeline is more of a utility pipeline that is run manually to build artifacts corresponding to a specific version and package them in s3 zip file. It does not publish anything to the MXNet website. It automates the job that is otherwise required to be done manually by the release manager.
It is an improvement over the pipeline that was used to build v1.7 artifacts: http://jenkins.mxnet-ci-dev.amazon-ml.com/job/docs/job/1.7-test/ which used regular slaves as well. This PR aims to check-in the pipeline code and add a stage to package artifacts in s3 zip file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r498683385
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,61 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
Review comment:
@waytrue17 we can skip archiving `full_website.tgz` as it is already stashed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r498513018
##########
File path: ci/jenkins/Jenkinsfile_website_release
##########
@@ -0,0 +1,67 @@
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// timeout in minutes
+max_time = 180
+
+node('utility') {
+ // Loading the utilities requires a node context unfortunately
+ checkout scm
+ utils = load('ci/Jenkinsfile_utils.groovy')
+ custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
+}
+
+utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')
Review comment:
> None the less, you are giving the pipeline write permissions. Our unrestricted slaves are set up to specifically not be able to persist anything.
Agreed, we can move the pipeline to jenkins-ci prod account, where we have access to restricted nodes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r496276413
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,20 @@ build_docs_beta() {
popd
}
+push_docs() {
+ version_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $version_name && tar -xzf full_website.tgz -C $version_name --strip-components 1
+ mv $version_name versions
Review comment:
check if folder already exists. if it does, exit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r496276504
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,20 @@ build_docs_beta() {
popd
}
+push_docs() {
+ version_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $version_name && tar -xzf full_website.tgz -C $version_name --strip-components 1
+ mv $version_name versions
+ zip -r9 versions-test.zip versions/.
Review comment:
change zip file name
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497662312
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,27 @@ build_docs_beta() {
popd
}
+push_docs() {
+ folder_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $folder_name && tar -xzf full_website.tgz -C $folder_name --strip-components 1
+ # check if folder_name already exists in versions
+ pushd versions
+ if [ -d "$folder_name" ]; then
+ echo "Folder $folder_name already exists in versions. Please double check the FOLDER_NAME variable in Jenkens pipeline"
+ exit 1
+ fi
+ popd
+ mv $folder_name versions
+ zip -r9 versions.zip versions/.
+ aws s3 cp versions.zip s3://mxnet-website-static-artifacts --acl bucket-owner-full-control
Review comment:
I passed the `public-read` to the s3 bucket to make the artifacts public.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r499036656
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,61 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
Review comment:
@marcoabreu removed archiving call, please review thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-706420343
Merging this as it helps automating an important part of website building and all the requested changes by @marcoabreu are addressed.
@marcoabreu - Thank you for you review and inputs. Since we were unable to reach you, and I see all your comments were addressed by @waytrue17 merging this PR.
Please feel free to comment later, if there are any open items and we can cover it in another PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r499036656
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,61 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
Review comment:
removed archiving call
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497947138
##########
File path: ci/jenkins/Jenkinsfile_website_release
##########
@@ -0,0 +1,67 @@
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// timeout in minutes
+max_time = 180
+
+node('utility') {
+ // Loading the utilities requires a node context unfortunately
+ checkout scm
+ utils = load('ci/Jenkinsfile_utils.groovy')
+ custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
+}
+
+utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')
Review comment:
@marcoabreu Just like @mseth10 mentioned, this pipeline will only be used to build the new website artifacts and upload them to a private S3 bucket for internal use.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] ys2843 commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
ys2843 commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-706434144
@waytrue17 It doesn't need to be port to master, only to branches that will be built as static artifact.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r496276563
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,59 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
Review comment:
we dont need this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497797953
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,27 @@ build_docs_beta() {
popd
}
+push_docs() {
+ folder_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $folder_name && tar -xzf full_website.tgz -C $folder_name --strip-components 1
+ # check if folder_name already exists in versions
+ pushd versions
+ if [ -d "$folder_name" ]; then
+ echo "Folder $folder_name already exists in versions. Please double check the FOLDER_NAME variable in Jenkens pipeline"
+ exit 1
+ fi
+ popd
+ mv $folder_name versions
+ zip -r9 versions.zip versions/.
+ aws s3 cp versions.zip s3://mxnet-website-static-artifacts --acl bucket-owner-full-control
Review comment:
We do not want to change the s3 bucket accessibility permissions with this PR. @ys2843 can you please confirm that `public-read` is the right permission for the bucket?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-702482830
@mxnet-bot run ci [unix-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r496360502
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,20 @@ build_docs_beta() {
popd
}
+push_docs() {
+ version_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $version_name && tar -xzf full_website.tgz -C $version_name --strip-components 1
+ mv $version_name versions
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497998263
##########
File path: ci/jenkins/Jenkinsfile_website_release
##########
@@ -0,0 +1,67 @@
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// timeout in minutes
+max_time = 180
+
+node('utility') {
+ // Loading the utilities requires a node context unfortunately
+ checkout scm
+ utils = load('ci/Jenkinsfile_utils.groovy')
+ custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
+}
+
+utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')
Review comment:
None the less, you are giving the pipeline write permissions. Our unrestricted slaves are set up to specifically not be able to persist anything.
Otherwise, somebody could go ahead, submit a PR which hijacks a pipeline. They could then go ahead and download illegal content from somewhere and host it in our s3 bucket. Do you want to be responsible for that?
Back then, the slaves were specifically set up with an instance profile which prohibited all these actions. You might want to engage in a security review to ensure everything is still safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-700203084
Hey @waytrue17 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [windows-cpu, clang, sanity, unix-cpu, centos-gpu, edge, centos-cpu, miscellaneous, unix-gpu, website, windows-gpu]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r496276413
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,20 @@ build_docs_beta() {
popd
}
+push_docs() {
+ version_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $version_name && tar -xzf full_website.tgz -C $version_name --strip-components 1
+ mv $version_name versions
Review comment:
check if version name folder already exists, if it does delete it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497748476
##########
File path: ci/jenkins/Jenkinsfile_website_release
##########
@@ -0,0 +1,67 @@
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// timeout in minutes
+max_time = 180
+
+node('utility') {
+ // Loading the utilities requires a node context unfortunately
+ checkout scm
+ utils = load('ci/Jenkinsfile_utils.groovy')
+ custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
+}
+
+utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')
Review comment:
Since this is a release pipeline, it must be run as restricted job on restricted slaves. Regular slaves are not permitted to have any publish access.
@leezu @aaronmarkham
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497662312
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,27 @@ build_docs_beta() {
popd
}
+push_docs() {
+ folder_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $folder_name && tar -xzf full_website.tgz -C $folder_name --strip-components 1
+ # check if folder_name already exists in versions
+ pushd versions
+ if [ -d "$folder_name" ]; then
+ echo "Folder $folder_name already exists in versions. Please double check the FOLDER_NAME variable in Jenkens pipeline"
+ exit 1
+ fi
+ popd
+ mv $folder_name versions
+ zip -r9 versions.zip versions/.
+ aws s3 cp versions.zip s3://mxnet-website-static-artifacts --acl bucket-owner-full-control
Review comment:
I passed the `public-read` to make the artifacts public.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r498517491
##########
File path: ci/jenkins/Jenkinsfile_website_release
##########
@@ -0,0 +1,67 @@
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// timeout in minutes
+max_time = 180
+
+node('utility') {
+ // Loading the utilities requires a node context unfortunately
+ checkout scm
+ utils = load('ci/Jenkinsfile_utils.groovy')
+ custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
+}
+
+utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')
Review comment:
@marcoabreu @mseth10 The pipeline has beed switched to restricted node and tested on jenkins-ci prod account.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy merged pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy merged pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r498687437
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,61 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
+ }
+ }
+ }
+ }]
+}
+
+// This is for uploading website artifacts to S3 bucket
+// Assumes you have run docs_full_website function
+def docs_upload_s3() {
+ return ['Upload artifacts to s3 bucket': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ if(env.FOLDER_NAME) {
+ utils.init_git()
+
+ unstash 'full_website'
+
Review comment:
```suggestion
utils.unpack_and_init('full_website', 'docs/_build/full_website.tgz')
```
@waytrue17 we can also make use of this utility fn here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r498683385
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,61 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
Review comment:
@waytrue17 let's skip archiving `full_website.tgz`, it is already stashed by `pack_lib` above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] samskalicky commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
samskalicky commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-704717179
@marcoabreu can you please review again now that your feedback has been addressed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-700850310
Hi @ys2843 Could you please review this PR? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-705864875
@marcoabreu - Looks like your review comments are addressed. Can you take a look at this and approve / have other feedback?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-705864875
@marcoabreu - Looks like your review comments are addressed. Can you take a look at this and approve / have other feedback?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-700324695
After this PR gets merged, modify Step 3.2.1 Point 5 in
https://cwiki.apache.org/confluence/display/MXNET/Release+Process#ReleaseProcess-3.2.UpdatetheMXNetwebsite
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
mseth10 commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-706439185
Hey @ys2843 , I thought we will need it for any release going forward, be it 1.9 or 2.0. How do we plan to build static artifacts for 2.0 once that branch is cut?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r498517491
##########
File path: ci/jenkins/Jenkinsfile_website_release
##########
@@ -0,0 +1,67 @@
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// timeout in minutes
+max_time = 180
+
+node('utility') {
+ // Loading the utilities requires a node context unfortunately
+ checkout scm
+ utils = load('ci/Jenkinsfile_utils.groovy')
+ custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
+}
+
+utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')
Review comment:
@marcoabrew @mseth10 The pipeline has beed switched to restricted node and tested on jenkins-ci prod account.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497947138
##########
File path: ci/jenkins/Jenkinsfile_website_release
##########
@@ -0,0 +1,67 @@
+// -*- mode: groovy -*-
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+// Jenkins pipeline
+// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
+
+// timeout in minutes
+max_time = 180
+
+node('utility') {
+ // Loading the utilities requires a node context unfortunately
+ checkout scm
+ utils = load('ci/Jenkinsfile_utils.groovy')
+ custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
+}
+
+utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')
Review comment:
@marcoabreu Just like @mseth10 mentioned, this pipeline will only be used to build the new website artifacts and upload them to a private S3 bucket for internal use. Sorry that the file name `Jenkinsfile_website_release` might be confusing. I've changed it to `Jenkinsfile_website_upload`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [v1.8.x] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r497662312
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,27 @@ build_docs_beta() {
popd
}
+push_docs() {
+ folder_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $folder_name && tar -xzf full_website.tgz -C $folder_name --strip-components 1
+ # check if folder_name already exists in versions
+ pushd versions
+ if [ -d "$folder_name" ]; then
+ echo "Folder $folder_name already exists in versions. Please double check the FOLDER_NAME variable in Jenkens pipeline"
+ exit 1
+ fi
+ popd
+ mv $folder_name versions
+ zip -r9 versions.zip versions/.
+ aws s3 cp versions.zip s3://mxnet-website-static-artifacts --acl bucket-owner-full-control
Review comment:
Yes, It should be accessible since I passed the `bucket-owner-full-control` to the s3 bucket. The uploaded file should have the same permissions as other files in the bucket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 commented on a change in pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
mseth10 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r496276563
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,59 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
Review comment:
we dont need this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on a change in pull request #19244: [WIP] Website release pipeline automation
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r496360408
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -1922,6 +1922,20 @@ build_docs_beta() {
popd
}
+push_docs() {
+ version_name=$1
+ set -ex
+ pip3 install --user awscli
+ export PATH=~/.local/bin:$PATH
+ pushd docs/_build
+ wget https://mxnet-website-static-artifacts.s3.us-east-2.amazonaws.com/versions.zip && unzip versions.zip && rm versions.zip
+ mkdir $version_name && tar -xzf full_website.tgz -C $version_name --strip-components 1
+ mv $version_name versions
+ zip -r9 versions-test.zip versions/.
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] waytrue17 commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
waytrue17 commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-706429927
> Yes, this needs to be backported to 1.x and master. @waytrue17 can you create those PRs?
Sure, I'll backport it to 1.x and master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#issuecomment-706420610
@waytrue17 / @mseth10 / @ys2843 => Should this be backported to 1.X branch please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #19244: [v1.8.x] Automate website artifacts uploading
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #19244:
URL: https://github.com/apache/incubator-mxnet/pull/19244#discussion_r498634763
##########
File path: ci/jenkins/Jenkins_steps.groovy
##########
@@ -1587,6 +1587,61 @@ def docs_jekyll() {
}]
}
+// This is for building the full website
+// Assumes you have run all of the docs generation functions
+def docs_full_website() {
+ return ['Build artifacts full_website.tgz': {
+ node(NODE_LINUX_CPU) {
+ ws('workspace/docs') {
+ timeout(time: max_time, unit: 'MINUTES') {
+ utils.init_git()
+
+ unstash 'jekyll-artifacts'
+ unstash 'c-artifacts'
+ unstash 'python-artifacts'
+ unstash 'r-artifacts'
+ unstash 'julia-artifacts'
+ unstash 'scala-artifacts'
+ unstash 'java-artifacts'
+ unstash 'clojure-artifacts'
+
+ utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
+ utils.pack_lib('full_website', 'docs/_build/full_website.tgz', false)
+
+ // archive so the publish pipeline can access the artifact
+ archiveArtifacts 'docs/_build/full_website.tgz'
Review comment:
Try to avoid archive artifacts as much as possible as the space on the jenkins master is limited. Instead, use stash to pass between stages and s3 to pass between pipelines. Same below
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org