You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/08 08:37:13 UTC
[GitHub] [pulsar] nicoloboschi opened a new pull request, #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
nicoloboschi opened a new pull request, #17541:
URL: https://github.com/apache/pulsar/pull/17541
### Motivation
1. For doc or cpp changes now the Pulsar CI workflow requires runners to just skip steps.
2. CPP builds run on different workflows and this can be optimized in a single workflow
3. MacOS build runner is created even if there are only doc changes
4. GO functions check can reuse the same runner
### Modifications
* Moved the check for doc changes to jobs level instead of steps. In case the condition of `if` returns false, no new runners will be spawned
* Moved all CPP builds to the same workflow
* Moved go functions checks to the same runner
* Moved MacOs build as step after Pulsar CI workflow
- [x] `doc-not-needed`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a diff in pull request #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
Posted by GitBox <gi...@apache.org>.
lhotari commented on code in PR #17541:
URL: https://github.com/apache/pulsar/pull/17541#discussion_r966040000
##########
.github/workflows/pulsar-ci.yaml:
##########
@@ -598,9 +593,6 @@ jobs:
- name: Schema
group: SCHEMA
- - name: Plugin
- group: PLUGIN
-
Review Comment:
Is this intentionally removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a diff in pull request #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
Posted by GitBox <gi...@apache.org>.
lhotari commented on code in PR #17541:
URL: https://github.com/apache/pulsar/pull/17541#discussion_r966044239
##########
.github/workflows/pulsar-ci.yaml:
##########
@@ -610,30 +602,135 @@ jobs:
- name: Pulsar IO
group: PULSAR_IO
+ - name: Sql
+ group: SQL
+
+ steps:
+ - name: checkout
+ uses: actions/checkout@v2
+
+ - name: Tune Runner VM
+ uses: ./.github/actions/tune-runner-vm
+
+ - name: Setup ssh access to build runner VM
+ # ssh access is enabled for builds in own forks
+ if: ${{ github.repository != 'apache/pulsar' }}
+ uses: ./.github/actions/ssh-access
+ with:
+ limit-access-to-actor: true
+
+ - name: Cache local Maven repository
+ uses: actions/cache@v2
+ with:
+ path: |
+ ~/.m2/repository/*/*/*
+ !~/.m2/repository/org/apache/pulsar
+ key: ${{ runner.os }}-m2-dependencies-all-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ${{ runner.os }}-m2-dependencies-core-modules-${{ hashFiles('**/pom.xml') }}
+ ${{ runner.os }}-m2-dependencies-core-modules-
+
+ - name: Set up JDK 17
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 17
+
+ - name: Install gh-actions-artifact-client.js
+ uses: apache/pulsar-test-infra/gh-actions-artifact-client/dist@master
+
+ - name: Restore maven build results from Github artifact cache
+ run: |
+ cd $HOME
+ $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh restore_tar_from_github_actions_artifacts pulsar-maven-repository-binaries
+
+ - name: Load docker image apachepulsar/pulsar-test-latest-version:latest from Github artifact cache
+ run: |
+ $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh docker_load_image_from_github_actions_artifacts pulsar-test-latest-version-image
+
+ - name: Run setup commands
+ if: ${{ matrix.setup }}
+ run: |
+ ${{ matrix.setup }}
+
+ - name: Run system test group '${{ matrix.group }}'
+ run: |
+ ./build/run_integration_group.sh ${{ matrix.group }}
+
+ - name: print JVM thread dumps when cancelled
+ if: cancelled()
+ run: $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh print_thread_dumps
+
+ - name: Aggregates all test reports to ./test-reports and ./surefire-reports directories
+ if: ${{ always() }}
+ uses: ./.github/actions/copy-test-reports
+
+ - name: Publish Test Report
+ uses: apache/pulsar-test-infra/action-junit-report@master
+ if: ${{ always() }}
+ with:
+ report_paths: 'test-reports/TEST-*.xml'
+ annotate_only: 'true'
+
+ - name: Upload container logs
+ uses: actions/upload-artifact@v3
+ if: ${{ !success() }}
+ continue-on-error: true
+ with:
+ name: System-${{ matrix.group }}-container-logs
+ path: tests/integration/target/container-logs
+ retention-days: 7
+
+ - name: Upload Surefire reports
+ uses: actions/upload-artifact@v3
+ if: ${{ !success() }}
+ with:
+ name: System-${{ matrix.name }}-surefire-reports
+ path: surefire-reports
+ retention-days: 7
+
+ - name: Wait for ssh connection when build fails
+ # ssh access is enabled for builds in own forks
+ uses: ./.github/actions/ssh-access
+ if: ${{ failure() && github.repository != 'apache/pulsar' }}
+ continue-on-error: true
+ with:
+ action: wait
+
+ flaky-system-tests:
Review Comment:
what is the exact reason for adding this? I would assume that it's possible to add matrix variables to cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #17541:
URL: https://github.com/apache/pulsar/pull/17541#issuecomment-1240522150
> Moved the check for doc changes to jobs level instead of steps. In case the condition of if returns false, no new runners will be spawned
That solution is possible only when the check produced by the skipped build job isn't required That's why the skipping solution is needed at the build step level. It shouldn't be done at such a fine grain level. A workflow where the last step is the required check would be a way to solve this so that the builds wouldn't have to be scheduled at all. This would also simplify the workflows since we could remove the step level skipping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on pull request #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on PR #17541:
URL: https://github.com/apache/pulsar/pull/17541#issuecomment-1240745429
I moved also the non required checks to the flaky workflow and now we have the latest jobs for Pulsar CI that will need to be put as required for merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a diff in pull request #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
Posted by GitBox <gi...@apache.org>.
lhotari commented on code in PR #17541:
URL: https://github.com/apache/pulsar/pull/17541#discussion_r966042062
##########
.github/workflows/pulsar-ci.yaml:
##########
@@ -762,4 +858,4 @@ jobs:
- name: Delete maven repository binaries from GitHub Actions Artifacts
run: |
Review Comment:
`if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}` is needed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
Posted by GitBox <gi...@apache.org>.
lhotari merged PR #17541:
URL: https://github.com/apache/pulsar/pull/17541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a diff in pull request #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
Posted by GitBox <gi...@apache.org>.
lhotari commented on code in PR #17541:
URL: https://github.com/apache/pulsar/pull/17541#discussion_r966065821
##########
.github/workflows/pulsar-ci.yaml:
##########
@@ -610,30 +602,135 @@ jobs:
- name: Pulsar IO
group: PULSAR_IO
+ - name: Sql
+ group: SQL
+
+ steps:
+ - name: checkout
+ uses: actions/checkout@v2
+
+ - name: Tune Runner VM
+ uses: ./.github/actions/tune-runner-vm
+
+ - name: Setup ssh access to build runner VM
+ # ssh access is enabled for builds in own forks
+ if: ${{ github.repository != 'apache/pulsar' }}
+ uses: ./.github/actions/ssh-access
+ with:
+ limit-access-to-actor: true
+
+ - name: Cache local Maven repository
+ uses: actions/cache@v2
+ with:
+ path: |
+ ~/.m2/repository/*/*/*
+ !~/.m2/repository/org/apache/pulsar
+ key: ${{ runner.os }}-m2-dependencies-all-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ${{ runner.os }}-m2-dependencies-core-modules-${{ hashFiles('**/pom.xml') }}
+ ${{ runner.os }}-m2-dependencies-core-modules-
+
+ - name: Set up JDK 17
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 17
+
+ - name: Install gh-actions-artifact-client.js
+ uses: apache/pulsar-test-infra/gh-actions-artifact-client/dist@master
+
+ - name: Restore maven build results from Github artifact cache
+ run: |
+ cd $HOME
+ $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh restore_tar_from_github_actions_artifacts pulsar-maven-repository-binaries
+
+ - name: Load docker image apachepulsar/pulsar-test-latest-version:latest from Github artifact cache
+ run: |
+ $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh docker_load_image_from_github_actions_artifacts pulsar-test-latest-version-image
+
+ - name: Run setup commands
+ if: ${{ matrix.setup }}
+ run: |
+ ${{ matrix.setup }}
+
+ - name: Run system test group '${{ matrix.group }}'
+ run: |
+ ./build/run_integration_group.sh ${{ matrix.group }}
+
+ - name: print JVM thread dumps when cancelled
+ if: cancelled()
+ run: $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh print_thread_dumps
+
+ - name: Aggregates all test reports to ./test-reports and ./surefire-reports directories
+ if: ${{ always() }}
+ uses: ./.github/actions/copy-test-reports
+
+ - name: Publish Test Report
+ uses: apache/pulsar-test-infra/action-junit-report@master
+ if: ${{ always() }}
+ with:
+ report_paths: 'test-reports/TEST-*.xml'
+ annotate_only: 'true'
+
+ - name: Upload container logs
+ uses: actions/upload-artifact@v3
+ if: ${{ !success() }}
+ continue-on-error: true
+ with:
+ name: System-${{ matrix.group }}-container-logs
+ path: tests/integration/target/container-logs
+ retention-days: 7
+
+ - name: Upload Surefire reports
+ uses: actions/upload-artifact@v3
+ if: ${{ !success() }}
+ with:
+ name: System-${{ matrix.name }}-surefire-reports
+ path: surefire-reports
+ retention-days: 7
+
+ - name: Wait for ssh connection when build fails
+ # ssh access is enabled for builds in own forks
+ uses: ./.github/actions/ssh-access
+ if: ${{ failure() && github.repository != 'apache/pulsar' }}
+ continue-on-error: true
+ with:
+ action: wait
+
+ flaky-system-tests:
Review Comment:
oh yes you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on a diff in pull request #17541: [ci] Reduce runners load for pulls that affect only doc or cpp changes
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on code in PR #17541:
URL: https://github.com/apache/pulsar/pull/17541#discussion_r966049332
##########
.github/workflows/pulsar-ci.yaml:
##########
@@ -610,30 +602,135 @@ jobs:
- name: Pulsar IO
group: PULSAR_IO
+ - name: Sql
+ group: SQL
+
+ steps:
+ - name: checkout
+ uses: actions/checkout@v2
+
+ - name: Tune Runner VM
+ uses: ./.github/actions/tune-runner-vm
+
+ - name: Setup ssh access to build runner VM
+ # ssh access is enabled for builds in own forks
+ if: ${{ github.repository != 'apache/pulsar' }}
+ uses: ./.github/actions/ssh-access
+ with:
+ limit-access-to-actor: true
+
+ - name: Cache local Maven repository
+ uses: actions/cache@v2
+ with:
+ path: |
+ ~/.m2/repository/*/*/*
+ !~/.m2/repository/org/apache/pulsar
+ key: ${{ runner.os }}-m2-dependencies-all-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ${{ runner.os }}-m2-dependencies-core-modules-${{ hashFiles('**/pom.xml') }}
+ ${{ runner.os }}-m2-dependencies-core-modules-
+
+ - name: Set up JDK 17
+ uses: actions/setup-java@v2
+ with:
+ distribution: 'temurin'
+ java-version: 17
+
+ - name: Install gh-actions-artifact-client.js
+ uses: apache/pulsar-test-infra/gh-actions-artifact-client/dist@master
+
+ - name: Restore maven build results from Github artifact cache
+ run: |
+ cd $HOME
+ $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh restore_tar_from_github_actions_artifacts pulsar-maven-repository-binaries
+
+ - name: Load docker image apachepulsar/pulsar-test-latest-version:latest from Github artifact cache
+ run: |
+ $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh docker_load_image_from_github_actions_artifacts pulsar-test-latest-version-image
+
+ - name: Run setup commands
+ if: ${{ matrix.setup }}
+ run: |
+ ${{ matrix.setup }}
+
+ - name: Run system test group '${{ matrix.group }}'
+ run: |
+ ./build/run_integration_group.sh ${{ matrix.group }}
+
+ - name: print JVM thread dumps when cancelled
+ if: cancelled()
+ run: $GITHUB_WORKSPACE/build/pulsar_ci_tool.sh print_thread_dumps
+
+ - name: Aggregates all test reports to ./test-reports and ./surefire-reports directories
+ if: ${{ always() }}
+ uses: ./.github/actions/copy-test-reports
+
+ - name: Publish Test Report
+ uses: apache/pulsar-test-infra/action-junit-report@master
+ if: ${{ always() }}
+ with:
+ report_paths: 'test-reports/TEST-*.xml'
+ annotate_only: 'true'
+
+ - name: Upload container logs
+ uses: actions/upload-artifact@v3
+ if: ${{ !success() }}
+ continue-on-error: true
+ with:
+ name: System-${{ matrix.group }}-container-logs
+ path: tests/integration/target/container-logs
+ retention-days: 7
+
+ - name: Upload Surefire reports
+ uses: actions/upload-artifact@v3
+ if: ${{ !success() }}
+ with:
+ name: System-${{ matrix.name }}-surefire-reports
+ path: surefire-reports
+ retention-days: 7
+
+ - name: Wait for ssh connection when build fails
+ # ssh access is enabled for builds in own forks
+ uses: ./.github/actions/ssh-access
+ if: ${{ failure() && github.repository != 'apache/pulsar' }}
+ continue-on-error: true
+ with:
+ action: wait
+
+ flaky-system-tests:
Review Comment:
yes but then the new step `pulsar-ci-checks-completed` needs these tests passing. Currently in the `.asf.yaml` they are not listed. So now it depends only on `system-tests` job and the result of this one doesn't impact the mergeablity of the pull
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org