You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/08/12 05:44:50 UTC
[GitHub] [flink] rmetzger opened a new pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
rmetzger opened a new pull request #13125:
URL: https://github.com/apache/flink/pull/13125
## What is the purpose of the change
Every night, we will deploy a SNAPSHOT version to Apache's snapshot maven repository. The Flink PMC got special credentials for that purpose.
We are also creating a Flink binary release every night. The binaries are provided as Azure artifacts.
We are re-using existing scripts for creating regular releases.
## Verifying this change
This is a test run of the two jobs: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8262&view=results
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger closed pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
rmetzger closed pull request #13125:
URL: https://github.com/apache/flink/pull/13125
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13125:
URL: https://github.com/apache/flink/pull/13125#discussion_r470533872
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
Review comment:
what we could maybe do is specify `task: CmdLine@2`, have the display name below, then the script block?
I think we can make these files much more readable if the display name is closer to the start of the block.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger edited a comment on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
rmetzger edited a comment on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-675040322
S3 Upload ✅
`task: CmdLine@2` ✅
renaming displayName: ⛔ (will do while merging)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-672622123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b6c7c2440191c0fba1a8ca796bbc15e70eeb049e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5433",
"triggerID" : "b6c7c2440191c0fba1a8ca796bbc15e70eeb049e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b6c7c2440191c0fba1a8ca796bbc15e70eeb049e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5433)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-672611771
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit b6c7c2440191c0fba1a8ca796bbc15e70eeb049e (Wed Aug 12 05:47:24 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #13125:
URL: https://github.com/apache/flink/pull/13125#discussion_r469326891
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+ export MVN="run_mvn"
+
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn --file pom.xml org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate -Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION' "
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true ./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ displayName: Build nightly
+ # deploy binary release
+ - task: PublishPipelineArtifact@1
+ displayName: Upload nightly
+ inputs:
+ targetPath: ./tools/releasing/release
+ artifact: nightly-release
+ - job: ${{parameters.stage_name}}_maven
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ timeoutInMinutes: 100 # 40 minutes per scala version + 20 buffer
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Upload snapshot
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+
+ cd tools
+ cat << EOF > deploy-settings.xml
+ <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
+ https://maven.apache.org/xsd/settings-1.0.0.xsd">
+ <servers>
+ <server>
+ <id>apache.snapshots.https</id>
+ <username>${MAVEN_DEPLOY_USER}</username>
+ <password>${MAVEN_DEPLOY_PASS}</password>
+ </server>
+ </servers>
+ <mirrors>
+ <mirror>
+ <id>google-maven-central</id>
Review comment:
This mirror is for downloading artifacts more reliably from Google's CDN, instead of the Apache Maven Central servers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-672761941
I generally want to reduce the number of scripts that we have around + using the release scripts for nightly and full releases seems elegant: We'll ensure that the release scripts are working during the release cycle.
But I can combine my desire with your suggestion and move the s3 upload part into the pipeline definition, because I generally agree that uploading to S3 is a good idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #13125:
URL: https://github.com/apache/flink/pull/13125#discussion_r471620280
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
Review comment:
Yep, that works. Once I have validated that the S3 upload works, I'll update the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13125:
URL: https://github.com/apache/flink/pull/13125#discussion_r469858082
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+ export MVN="run_mvn"
+
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn --file pom.xml org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate -Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION' "
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true ./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ displayName: Build nightly
Review comment:
```suggestion
displayName: Build snapshot binary release
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-675040322
S3 Upload ✅
`task: CmdLine@2` ✅
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13125:
URL: https://github.com/apache/flink/pull/13125#discussion_r472068621
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ displayName: Cache Maven local repo
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ continueOnError: true
+ - task: CmdLine@2
+ displayName: Build Flink
+ inputs:
+ script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+ export MVN="run_mvn"
+
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION' "
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true ./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ cd ..
+ - task: CmdLine@2
+ displayName: Upload artifacts to S3
+ inputs:
+ script: |
+ source ./tools/ci/deploy_nightly_to_s3.sh
+
+ upload_to_s3 ./tools/releasing/release
+ env:
+ ARTIFACTS_S3_BUCKET: $(ARTIFACTS_S3_BUCKET)
+ ARTIFACTS_AWS_ACCESS_KEY_ID: $(ARTIFACTS_AWS_ACCESS_KEY_ID)
+ ARTIFACTS_AWS_SECRET_ACCESS_KEY: $(ARTIFACTS_AWS_SECRET_ACCESS_KEY)
+ # Activate this to publish the binary release as a pipeline artifact on Azure
+ #- task: PublishPipelineArtifact@1
+ # displayName: Upload snapshot binary release
+ # inputs:
+ # targetPath: ./tools/releasing/release
+ # artifact: nightly-release
+ - job: ${{parameters.stage_name}}_maven
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ timeoutInMinutes: 100 # 40 minutes per scala version + 20 buffer
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
Review comment:
```suggestion
- task: Cache@2
displayName: Cache Maven local repo
inputs:
key: $(CACHE_KEY)
restoreKeys: $(CACHE_FALLBACK_KEY)
path: $(MAVEN_CACHE_FOLDER)
continueOnError: true
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13125:
URL: https://github.com/apache/flink/pull/13125#discussion_r469190780
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+ export MVN="run_mvn"
+
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn --file pom.xml org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate -Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION' "
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true ./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ displayName: Build nightly
+ # deploy binary release
+ - task: PublishPipelineArtifact@1
+ displayName: Upload nightly
Review comment:
```suggestion
- task: PublishPipelineArtifact@1
displayName: Upload snapshot binary release
```
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+ export MVN="run_mvn"
+
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn --file pom.xml org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate -Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION' "
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true ./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ displayName: Build nightly
+ # deploy binary release
+ - task: PublishPipelineArtifact@1
+ displayName: Upload nightly
+ inputs:
+ targetPath: ./tools/releasing/release
+ artifact: nightly-release
+ - job: ${{parameters.stage_name}}_maven
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ timeoutInMinutes: 100 # 40 minutes per scala version + 20 buffer
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Upload snapshot
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+
+ cd tools
+ cat << EOF > deploy-settings.xml
+ <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
+ https://maven.apache.org/xsd/settings-1.0.0.xsd">
+ <servers>
+ <server>
+ <id>apache.snapshots.https</id>
+ <username>${MAVEN_DEPLOY_USER}</username>
+ <password>${MAVEN_DEPLOY_PASS}</password>
+ </server>
+ </servers>
+ <mirrors>
+ <mirror>
+ <id>google-maven-central</id>
Review comment:
what is this mirror for? If it does what I think it is (mirroring the artifacts to maven central where they are publicly accessible to all users), then it should be removed.
##########
File path: tools/azure-pipelines/build-apache-repo.yml
##########
@@ -74,6 +74,11 @@ stages:
dependsOn: [] # depending on an empty array makes the stages run in parallel
condition: or(eq(variables['Build.Reason'], 'Schedule'), eq(variables['MODE'], 'nightly'))
jobs:
+ - template: build-nightly-dist.yml
+ parameters:
+ stage_name: cron_nightly_dist
Review comment:
```suggestion
stage_name: cron_snapshot_deployment
```
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+ export MVN="run_mvn"
+
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn --file pom.xml org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate -Dexpression=project.version -q -DforceStdout)
Review comment:
```suggestion
export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
```
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
Review comment:
is it a hard requirement that the script is the first item? If not, then having the displayName as the first item would eliminate the requirement of needing a comment to explain what it does.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-672622123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b6c7c2440191c0fba1a8ca796bbc15e70eeb049e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5433",
"triggerID" : "b6c7c2440191c0fba1a8ca796bbc15e70eeb049e",
"triggerType" : "PUSH"
}, {
"hash" : "c24b96bd51f97fb3922d61dd7338a5dd5ff9036b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c24b96bd51f97fb3922d61dd7338a5dd5ff9036b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b6c7c2440191c0fba1a8ca796bbc15e70eeb049e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5433)
* c24b96bd51f97fb3922d61dd7338a5dd5ff9036b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13125:
URL: https://github.com/apache/flink/pull/13125#discussion_r469858514
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+ export MVN="run_mvn"
+
+ export RELEASE_VERSION=$(MVN_RUN_VERBOSE=false run_mvn --file pom.xml org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate -Dexpression=project.version -q -DforceStdout)
+ echo "Determined RELEASE_VERSION as '$RELEASE_VERSION' "
+ cd tools
+ MVN_RUN_VERBOSE=true SKIP_GPG=true ./releasing/create_binary_release.sh
+ echo "Created files:"
+ find ./releasing/release
+ displayName: Build nightly
+ # deploy binary release
+ - task: PublishPipelineArtifact@1
+ displayName: Upload nightly
+ inputs:
+ targetPath: ./tools/releasing/release
+ artifact: nightly-release
+ - job: ${{parameters.stage_name}}_maven
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ timeoutInMinutes: 100 # 40 minutes per scala version + 20 buffer
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Upload snapshot
+ - script: |
+ source ./tools/ci/maven-utils.sh
+ setup_maven
+ run_mvn -version
+
+ cd tools
+ cat << EOF > deploy-settings.xml
+ <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
+ https://maven.apache.org/xsd/settings-1.0.0.xsd">
+ <servers>
+ <server>
+ <id>apache.snapshots.https</id>
+ <username>${MAVEN_DEPLOY_USER}</username>
+ <password>${MAVEN_DEPLOY_PASS}</password>
+ </server>
+ </servers>
+ <mirrors>
+ <mirror>
+ <id>google-maven-central</id>
Review comment:
well, that makes sense, and in hindsight was rather obvious.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-672622123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b6c7c2440191c0fba1a8ca796bbc15e70eeb049e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5433",
"triggerID" : "b6c7c2440191c0fba1a8ca796bbc15e70eeb049e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b6c7c2440191c0fba1a8ca796bbc15e70eeb049e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5433)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-675610845
Thanks a lot for your review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13125:
URL: https://github.com/apache/flink/pull/13125#issuecomment-672622123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b6c7c2440191c0fba1a8ca796bbc15e70eeb049e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b6c7c2440191c0fba1a8ca796bbc15e70eeb049e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b6c7c2440191c0fba1a8ca796bbc15e70eeb049e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #13125: [FLINK-18643] Deploy Flink snapshot build with nightly CI job
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #13125:
URL: https://github.com/apache/flink/pull/13125#discussion_r469329511
##########
File path: tools/azure-pipelines/build-nightly-dist.yml
##########
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jobs:
+ - job: ${{parameters.stage_name}}_binary
+ pool:
+ vmImage: 'ubuntu-16.04'
+ container: flink-build-container
+ workspace:
+ clean: all
+ steps:
+ - task: Cache@2
+ inputs:
+ key: $(CACHE_KEY)
+ restoreKeys: $(CACHE_FALLBACK_KEY)
+ path: $(MAVEN_CACHE_FOLDER)
+ displayName: Cache Maven local repo
+ continueOnError: true
+ # Create binary release
+ - script: |
Review comment:
I don't think there's anything we can do about it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org