You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/23 10:39:26 UTC
[GitHub] [pulsar] lhotari opened a new pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
lhotari opened a new pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819
### Motivation
Improve Pulsar CI:
- Reduce GitHub Action Runner resource consumption of Pulsar PR builds
- Currently, Pulsar GitHub Actions workflows are consuming the majority of the shared pool of resources allocated for github.com/apache projects
- Running the GitHub Actions workflows for a single PR to Pulsar consumes about 18-20 hours of GitHub Actions Runner VM time. This is too much.
- Reduce lead times for Pull Request feedback by speeding up builds
- Speeds up Pulsar development
- Improves developer productivity since waiting times are reduced
- Since PR feedback is faster, developers can be comfortable submitting more granular pull requests.
- When development cycle is faster, it is easier to keep the pull request queue shorter. This has several benefits since when PRs are handled quickly, there are fewer chances for pull requests to divert from the master branch. It also reduces merge conflicts and the time wasted in resolving merge conflicts.
- Better usability and access to test reports
- Less time is spent in looking for the reason why a build failed
### Modifications
- The design goal has been to keep the build content as the same as before the refactoring. The same tests are run, but in more effective ways. This refactoring doesn't make changes to the way how test retries are handled.
- Combine most of the Pulsar CI workflows into a single workflow called "Pulsar CI"
- The workflows that benefit of the aggregation have been chosen.
- the modifications reuse binary artifacts in the workflow and this reduces the resource consumption.
- Pulsar core modules jar files are built once and reused.
- Pulsar docker images are built once and reused
- GitHub Actions cache is used to share the files. The capacity of GitHub Actions cache is 10GB which is scoped to the developer who opens the pull request. This means that there's plenty of disk space for PR builds (10GB for each developer).
- Integration tests are categorized into "integration tests" and "system tests"
- A slimmer docker image `apachepulsar/java-test-image:latest` is used to run the integration tests that don't depend on Pulsar Python client, Tiered storage drivers, Pulsar SQL or Pulsar Connectors.
- The previous `apachepulsar/pulsar-test-latest-version:latest` image is used to run the integration tests that are categorized as "system tests".
- The benefit of this split is that the java-test-image builds in about 6 minutes and can start the downstream integration test jobs after this. This results in faster developer feedback.
- For debugging builds, there's configuration for exposing ssh shell access to each Build VM to the user who triggered the build ("github actor"). The ssh access is authenticated with the SSH key that the user has registered in GitHub.
- ssh access is only active in own forks. It is not enabled in `apache/pulsar` because of security concerns.
- A developer can open a PR to their own fork (for example with a single command with GH cli `gh pr create --repo=githubusername/pulsar --base master --head "$(git branch --show-current)" -f`) to run the build with ssh access enabled.
- ssh access is active for the duration of the build. If the build fails, the build waits 5 minutes for a developer to connect to investigate the problem. (this behavior is not enabled in `apache/pulsar`)
The SSH shell access feature will make it easier to debug CI issues which don't get resolved with the information in the GitHub Actions UI. This is an important capability to have available whenever there are problems. As described above, the configuration requires to run the build in a developer's personal fork of the pulsar repository to activate the feature.
- Fix broken configuration in `.github/actions/tune-runner-vm/action.yml` which was broken with PR #13252.
- The makes Linux kernel's vm swappiness setting effectively `1` for all cgroups.
- Helps prevent swapping when the VM is running low on memory.
- Improve test reporting by the use of https://github.com/dorny/test-reporter . The test reports get attached to the wrong workflow because of a GitHub Actions limitation. That reduces the usability since the test reports are harder to find. test-reporter renders the Junit XML files to the GitHub Actions UI.
- Improve test reporting by adding warning annotations about the test statistics.
- not really warnings, but GitHub Actions doesn't seem to allow info annotations from shell scripts.
- Use GitHub Action built-in feature to cancel duplicate build jobs:
```
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
```
- a new push to a PR will trigger a new job and this feature will be used to cancel the previous build which is obsolete
- this solution might be more effective than the current solution to cancel duplicate jobs
### Additional Context
The work in this PR was mainly done last year while working on a proof-of-concept of the GitHub Actions refactoring.
There's a Google document [[Discuss] PIP Changes to GitHub Actions based Pulsar CI](https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry) which describes details about some technical solutions. There's also an [email thread on the dev mailing list](https://lists.apache.org/thread/ra2fcf7b973448bb51e00aceaeed06433e8d886270b0f0db0c80d4e0c@%3Cdev.pulsar.apache.org%3E).
The showstopper a year ago was the lack of being able to re-run a single failed job in a larger workflow.
GitHub has since then delivered this feature and no showstoppers are present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1076520205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1076229398
> IIUC now the pulsarbot is not needed anymore and you have to enter the workflow run and re-run the single job, is it ?
@nicoloboschi It's needed. This doesn't merge all the workflows to one. It's the majority of the workflows that are merged.
The bot is needed so that non-committers can restart failed jobs. It might require a tweak to pulsarbot so that it uses the GitHub Actions API to start just the failed jobs in a workflow and not the complete workflow and all jobs in it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1081102755
example of storing a docker image: https://github.com/apache/pulsar/runs/5726757372?check_suite_focus=true#step:11:23 .
example of restoring docker image: https://github.com/apache/pulsar/runs/5726913967?check_suite_focus=true#step:10:37
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r837573728
##########
File path: .github/actions/ssh-access/action.yml
##########
@@ -0,0 +1,147 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: ssh access
+description: Sets up SSH access to build VM with upterm
+inputs:
+ action:
+ description: |
+ Action to perform: options are "start" and "wait"
+ "start" will install, configure and start upterm.
+ "wait" will wait until a connection is established to upterm and will continue to wait until the session is closed.
+ required: false
+ default: 'start'
+ limit-access-to-actor:
+ description: 'If only the public SSH keys of the user triggering the workflow should be authorized'
+ required: false
+ default: 'false'
+ limit-access-to-users:
+ description: 'If only the public SSH keys of the listed GitHub users should be authorized. Comma separate list of GitHub user names.'
+ required: false
+ default: ''
+ secure-access:
+ description: |
+ Set to false for allowing public access when limit-access-to-actor and limit-access-to-users are unset.
+ required: false
+ default: 'true'
+ timeout:
+ description: 'When action=wait, the timeout in seconds to wait for the user to connect'
+ required: false
+ default: '300'
+runs:
+ using: composite
+ steps:
+ - run: |
+ if [[ "${{ inputs.action }}" == "start" ]]; then
+ echo "::group::Installing upterm & tmux"
+ if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+ # install upterm
+ curl -sL https://github.com/owenthereal/upterm/releases/download/v0.7.6/upterm_linux_amd64.tar.gz | tar zxvf - -C /tmp upterm && sudo install /tmp/upterm /usr/local/bin/ && rm -rf /tmp/upterm
Review comment:
Nit: it looks like upterm releases updates frequently, but we're pinned to 0.7.6 on linux. I don't see a good solution for downloading the latest version to linux, though. I've subscribed to release and security notifications for https://github.com/owenthereal/upterm. I wonder if that project would consider making their binaries available through another mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1082714042
I made one more change: I increased the GitHub Actions Artifact retention period to 3 days for the intermediate build artifacts which are retained if a build job fails. The intermediate build artifacts are the Pulsar maven repository artifacts for the current build and the 2 different docker images used for integration and system tests. These consume about 2.5GB of disk space in total.
It's possible to rerun individual failed jobs when the build artifacts are available. If the artifacts have already been expired, the complete workflow can be rerun by closing and reopening the PR or by rebasing the PR.
The reason to minimize the retention period is cost. It costs $0.25 per GB for a total month of retention. The retention of 2.5GB for 3 days costs about 2.5 * $0.25 / 30 * 3= $0.0625 . That's 6 cents. I would assume that Apache has a certain amount of donated credits from GitHub to run GitHub Actions.
For personal accounts, there is 500MB free monthly credit. This means that you can keep 500MB stored for a month. The storage cost is calculated hourly and it means that you could store 500MB*30 for 1 day if you wish. After the free credit is consumed, GitHub Actions will be disabled until the next billing period starts or the user purchases more credits.
For GitHub Actions, the log file retention consumes the same storage quota. The usage of GitHub Actions Artifacts doesn't introduce anything new in that sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1081102755
example of storing a docker image: https://github.com/apache/pulsar/runs/5726757372?check_suite_focus=true#step:11:23 .
```
Saving docker image apachepulsar/java-test-image:latest with name pulsar-java-test-image.zst in GitHub Actions Artifacts
Artifact name is valid!
0:00:05
0:00:10
256MiB [11.7MiB/s]
0:00:34
256MiB [11.7MiB/s]
0:00:34
0:00:58 512MiB [11.3MiB/s]
512MiB [11.3MiB/s]
0:00:58
0:00:58
527MiB [11.5MiB/s]
```
example of restoring docker image: https://github.com/apache/pulsar/runs/5726913967?check_suite_focus=true#step:10:37
```
Loading docker image from name pulsar-java-test-image.zst in GitHub Actions Artifacts
Total number of files that will be downloaded: 3
Total file count: 3 ---- Processed file #0 (0.0%)
Total file count: 3 ---- Processed file #0 (0.0%)
Total file count: 3 ---- Processed file #0 (0.0%)
Total file count: 3 ---- Processed file #0 (0.0%)
0.00 B 0:00:05 [0.00 B/s]
Total file count: 3 ---- Processed file #0 (0.0%)
Total file count: 3 ---- Processed file #0 (0.0%)
Total file count: 3 ---- Processed file #0 (0.0%)
Total file count: 3 ---- Processed file #0 (0.0%)
Total file count: 3 ---- Processed file #0 (0.0%)
0.00 B 0:00:10 [0.00 B/s]
Total file count: 3 ---- Processed file #0 (0.0%)
Total file count: 3 ---- Processed file #1 (33.3%)
Total file count: 3 ---- Processed file #1 (33.3%)
Total file count: 3 ---- Processed file #1 (33.3%)
Total file count: 3 ---- Processed file #1 (33.3%)
256MiB 0:00:15 [17.0MiB/s]
Total file count: 3 ---- Processed file #1 (33.3%)
Total file count: 3 ---- Processed file #1 (33.3%)
Total file count: 3 ---- Processed file #1 (33.3%)
Total file count: 3 ---- Processed file #1 (33.3%)
Total file count: 3 ---- Processed file #1 (33.3%)
256MiB 0:00:20 [12.8MiB/s]
Total file count: 3 ---- Processed file #1 (33.3%)
Total file count: 3 ---- Processed file #2 (66.6%)
527MiB 0:00:23 [22.8MiB/s]
Loaded image: apachepulsar/java-test-image:latest
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1076608600
I have created a separate PR to make pulsarbot support the new GitHub Actions feature of rerunning failed jobs (instead of all jobs in a workflow): https://github.com/apache/pulsar-test-infra/pull/27 /cc @nicoloboschi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r833123969
##########
File path: pulsar-io/docs/pom.xml
##########
@@ -82,6 +82,16 @@
<artifactId>pulsar-io-debezium-postgres</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review comment:
yes. Builds are made in a slightly different way and the maven build must have correct dependencies for maven's `-am` parameter to work.
In the first step, only `core-modules` are built and shared. Other modules are built on demand and that's why dependencies must be correctly specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r835059062
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
I have also asked the question in https://github.com/actions/cache/discussions/776 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r836789661
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
@yaalsn The gh-actions-artifact-client is now integration in the Pulsar CI refactoring. GitHub Actions Artifacts are used now instead of the GitHub Actions Cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1082714042
I made one more change: I increased the GitHub Actions Artifact retention period to 3 days for the intermediate build artifacts which are retained if a build job fails. The intermediate build artifacts are the Pulsar maven repository artifacts for the current build and the 2 different docker images used for integration and system tests. These consume about 2.5GB of disk space in total.
It's possible to rerun individual failed jobs when the build artifacts are available. If the artifacts have already been expired, the complete workflow can be rerun by closing and reopening the PR or by rebasing the PR.
The reason to minimize the retention period is cost. It costs $0.25 per GB for a total month of retention. The retention of 2.5GB for 3 days costs about 2.5 * $0.25 / 30 * 3= $0.0625 . That's 6 cents. I would assume that Apache has a certain amount of donated credits from GitHub to run GitHub Actions.
For personal accounts, there is 500MB free monthly credit. This means that you can keep 500MB stored for a month. The storage cost is calculated hourly and it means that you could store 500MB*30 for 1 day if you wish. After the free credit is consumed, GitHub Actions will be disabled until the next billing period starts or the user purchases more credits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r835799558
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
I'm writing a command line client for GitHub Actions cache so that it would integrate with the refactored build. The current work-in-progress is at https://github.com/lhotari/gh-actions-artifact-client . It uses the official GitHub Actions javascript npm modules for interacting with GitHub Actions Artifact backend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari merged pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1082720440
The reason to use 3 days for retention is convenience. If your build job fails on Friday, you can still retry it on Monday. Retries are currently relevant in Pulsar builds since there's a high number of flaky tests and retries are used so that flaky tests don't block progress. When a developer encounters a flaky test, it should always be checked whether it has already [been reported as a GitHub issue](https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc). If not, the developer [should report it](https://github.com/apache/pulsar/issues/new/choose).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yaalsn commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
yaalsn commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r835824710
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
GitHub Actions Artifact has a retention period option, the artifacts could be cleaned after all the workflow is finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari removed a comment on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari removed a comment on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1076466027
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1076225813
Since the workflow changes, the required checks change too. The `.asf.yaml` file is used to update the required checks. The changes are included in this PR, but it might require merging that file separately so that this PR could be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r835381596
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
I received a reply. https://github.community/t/how-does-the-github-actions-cache-size-limit-work-for-pull-requests/240546/6 . The 10GB limit is indeed in the scope of apache/pulsar repository. That is a problem since concurrent builds would hit the limit and cause frequent cache misses.
I'm now working on replacing the GitHub Actions Cache with the usage of [GitHub Actions Artifacts which is designed for sharing data between jobs in a workflow](https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts).
I'm doing a proof-of-concept of a command line interface to GitHub Actions Artifacts so that it would be possible to use integrate with `docker save` and `docker load` in the same way that I have done in the current refactored workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] yaalsn commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
yaalsn commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r833181215
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
@lhotari I am sorry but I still want to confirm with you that if this `http-cache-action` uses `github action cache`, just like this one https://github.com/actions/cache. And It seems that every repository has a 10GB cache limit total not every PR.
<img width="990" alt="image" src="https://user-images.githubusercontent.com/10069311/159694970-a8f7f79e-5a93-4576-8a57-225933416d1a.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1081834114
> great work
>
> I wonder when we can commit this.
Thanks! I believe this is a low risk change and we should go forward and commit it asap. There's no benefit in postponing it any further.
>
> I would send a final message to dev and if no one objects we can merge the patch (Lazy consensus)
+1. I'll do that. I've sent out many messages to the mailing list and Pulsar Slack so a short notice should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1082786305
There was a few last minute changes which required me to change the `.asf.yaml` once again. That's in #14944 .
I had forgotten `-Dmaven.test.failure.ignore=true -DtestRetryCount=0` as part of `MAVEN_OPTS` which made all test jobs pass always. I used that during the development of the workflow to speed up testing since flaky tests would cause retries and delay testing. After fixing that issue, I noticed that the Pulsar Functions integration tests and Schema integration tests require the Pulsar python client which is not available in the "CI - Integration - *" tests in the new build since it uses a Java only docker image which is faster to build. The "CI - System - *" build jobs use the ordinary `pulsar-test-latest-version-image` which has been used before.
The reason to run some integration tests with the Java only docker image is the benefit of getting earlier feedback. Building the `pulsar-test-latest-version-image` takes over 25 minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r833990274
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
@yaalsn Each user has 10 GB for pull requests in GitHub Actions Cache. That's part of the security model. You cannot make updates to the upstream repository's cache where you open a PR. It's hierarchical internally. Updates are only made in the PR owner's cache and impact it's quota.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari removed a comment on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari removed a comment on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1076506864
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1081074119
The PR has been revisited to use GitHub Actions Artifacts which is meant to be used for sharing data across jobs in a workflow.
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r833117410
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
is this port documented somewhere ?
##########
File path: pulsar-io/docs/pom.xml
##########
@@ -82,6 +82,16 @@
<artifactId>pulsar-io-debezium-postgres</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review comment:
is this related to the pull ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r833128262
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
Here's the documentation: https://github.com/cirruslabs/http-cache-action#port
The port 12321 is used for the http-cache-action which is used to access GitHub Actions cache with a HTTP interface.
The GitHub Actions cache doesn't currently have a CLI for storing arbitrary binary streams. That feature is needed in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari closed pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r837701416
##########
File path: .github/actions/ssh-access/action.yml
##########
@@ -0,0 +1,147 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: ssh access
+description: Sets up SSH access to build VM with upterm
+inputs:
+ action:
+ description: |
+ Action to perform: options are "start" and "wait"
+ "start" will install, configure and start upterm.
+ "wait" will wait until a connection is established to upterm and will continue to wait until the session is closed.
+ required: false
+ default: 'start'
+ limit-access-to-actor:
+ description: 'If only the public SSH keys of the user triggering the workflow should be authorized'
+ required: false
+ default: 'false'
+ limit-access-to-users:
+ description: 'If only the public SSH keys of the listed GitHub users should be authorized. Comma separate list of GitHub user names.'
+ required: false
+ default: ''
+ secure-access:
+ description: |
+ Set to false for allowing public access when limit-access-to-actor and limit-access-to-users are unset.
+ required: false
+ default: 'true'
+ timeout:
+ description: 'When action=wait, the timeout in seconds to wait for the user to connect'
+ required: false
+ default: '300'
+runs:
+ using: composite
+ steps:
+ - run: |
+ if [[ "${{ inputs.action }}" == "start" ]]; then
+ echo "::group::Installing upterm & tmux"
+ if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+ # install upterm
+ curl -sL https://github.com/owenthereal/upterm/releases/download/v0.7.6/upterm_linux_amd64.tar.gz | tar zxvf - -C /tmp upterm && sudo install /tmp/upterm /usr/local/bin/ && rm -rf /tmp/upterm
Review comment:
The ssh-access solution is only used in forks for debugging builds so this isn't a critical function. Some previous upterm releases have contained breaking changes so that's one additional reason to use a fixed version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1083200635
Merging the PR was blocked by #14951 . The Pulsar SQL tests stopped passing after a Java upgrade to 11.0.14.1 . To unblock CI, I have disabled Pulsar SQL integration tests in this PR. That's a temporary workaround until #14951 is resolved.
Pulsar SQL doesn't work with Java 11.0.14.1 version. Ubuntu got the 11.0.14.1 update today and that's why it happened to break just now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1083200635
Merging the PR is blocked by https://github.com/apache/pulsar/issues/14951 .
Pulsar SQL doesn't work with Java 11.0.14.1 version. Ubuntu got the 11.0.14.1 update today and that's why it happened to break just now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r836523893
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
> GitHub Actions Artifact has a retention period option, the artifacts could be cleaned after all the workflow is finished.
Yes. I think it's sufficient to let the retention period to take care of the clean up. I now have https://github.com/lhotari/gh-actions-artifact-client working and I tested it with a 600MB file. The performance is pretty good. About 11MB/s for writing and over 23 MB/s for reading (with limited testing). There's a chance to improve the performance further by adding some concurrency.
I'll start working on the changes to integrate it to the Pulsar CI refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1076563705
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r833122250
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
> IIUC now the pulsarbot is not needed anymore and you have to enter the workflow run and re-run the single job, is it ?
It's needed. This doesn't merge all the workflows to one. It's the majority of the workflows that are merged.
The bot is needed so that non-committers can restart failed jobs. It might require a tweak to pulsarbot so that it uses the GitHub Actions API to start just the failed jobs in a workflow and not the complete workflow and all jobs in it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r833123969
##########
File path: pulsar-io/docs/pom.xml
##########
@@ -82,6 +82,16 @@
<artifactId>pulsar-io-debezium-postgres</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review comment:
yes. Builds are made in a slightly different way and the maven build must have correct dependencies for `-am` to work.
In the first step, only `core-modules` are built and shared. Other modules are built on demand and that's why dependencies must be correctly specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#issuecomment-1083111044
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r837736566
##########
File path: .github/actions/ssh-access/action.yml
##########
@@ -0,0 +1,147 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: ssh access
+description: Sets up SSH access to build VM with upterm
+inputs:
+ action:
+ description: |
+ Action to perform: options are "start" and "wait"
+ "start" will install, configure and start upterm.
+ "wait" will wait until a connection is established to upterm and will continue to wait until the session is closed.
+ required: false
+ default: 'start'
+ limit-access-to-actor:
+ description: 'If only the public SSH keys of the user triggering the workflow should be authorized'
+ required: false
+ default: 'false'
+ limit-access-to-users:
+ description: 'If only the public SSH keys of the listed GitHub users should be authorized. Comma separate list of GitHub user names.'
+ required: false
+ default: ''
+ secure-access:
+ description: |
+ Set to false for allowing public access when limit-access-to-actor and limit-access-to-users are unset.
+ required: false
+ default: 'true'
+ timeout:
+ description: 'When action=wait, the timeout in seconds to wait for the user to connect'
+ required: false
+ default: '300'
+runs:
+ using: composite
+ steps:
+ - run: |
+ if [[ "${{ inputs.action }}" == "start" ]]; then
+ echo "::group::Installing upterm & tmux"
+ if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+ # install upterm
+ curl -sL https://github.com/owenthereal/upterm/releases/download/v0.7.6/upterm_linux_amd64.tar.gz | tar zxvf - -C /tmp upterm && sudo install /tmp/upterm /usr/local/bin/ && rm -rf /tmp/upterm
Review comment:
Sounds good, thanks for the extra context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r834103121
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
In apache/pulsar builds, there will be the 10GB limit for GitHub Actions cache. The cache is a LRU cache based on the [description in actions/cache README](https://github.com/actions/cache#cache-limits):
> A repository can have up to 10GB of caches. **Once the 10GB limit is reached, older caches will be evicted based on when the cache was last accessed.** Caches that are not accessed within the last week will also be evicted.
This behavior is optimal. Each build will temporarily create about 2.5GB of cache entries. Since the needed cache entries are accessed frequently, this will ensure that important caches (the maven dependencies caches) aren't likely to be dropped. maven dependencies caches are updated with a scheduled job once a day: https://github.com/apache/pulsar/blob/master/.github/workflows/ci-maven-cache-update.yaml
The 10GB limit could cause issues if there are too many concurrent build jobs for apache/pulsar. That problem can be addressed by falling back to building the binaries if there's a cache miss. That solution could be added later if problems occur. GitHub actions has also concurrency limits which are added in this PR. A more recent build will cancel a build that started earlier for a given branch. In the context of apache/pulsar, the builds will be pushes to master and branch-* branches. It would be a very rare case where there's more than 4 branches that are built at the same time. It's possible, but if that happens, the build can be improved to handle cache misses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r833992212
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
the https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit# doc also explains this.
> GitHub Actions Cache is restricted to the scope of the forked repository in pull requests to prevent security issues around cache poisoning. The additional benefit is that the total capacity of caches is beyond 10GB since the cache is not shared between forks. Each fork has a quota of 10GB. The GitHub Actions Cache can be used as a general purpose file storage to share the build artifacts as files between the stages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r834001248
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
I started a thread in GitHub Community support forum to confirm this: https://github.community/t/how-does-the-github-actions-cache-size-limit-work-for-pull-requests/240546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #14819: [CI] Refactor GitHub Actions based Pulsar CI
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r834001248
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify -pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5 | curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @- http://localhost:12321/${cachekey}"
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5 | unzstd | docker load"
Review comment:
I started a thread in GitHub Community support forum to clarify this: https://github.community/t/how-does-the-github-actions-cache-size-limit-work-for-pull-requests/240546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org