You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/07/07 09:50:56 UTC
[GitHub] [pulsar] Anonymitaet opened a new issue, #16439: [improve][workflow] Optimize CI/CD
Anonymitaet opened a new issue, #16439:
URL: https://github.com/apache/pulsar/issues/16439
The existing CI/CD workflow can be optimized for the following aspects:
### Issue 1
Compared with other projects' workflows, Pulsar has many workflows and the [list](https://github.com/apache/pulsar/actions) is long.
Abandoned workflows should be hidden and the same workflow should be combined together to make the list looks clearer and tidy.
### Issue 2
Shall we check if there are any duplicated or unreasonable workflows?
They can be deleted to release some resources or optimized.
🔹🔹🔹
@maxsxu @yaalsn thanks for your suggestions!
Any other thoughts on this? Anyone would like to help Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1182816937
@Anonymitaet Please don't do any GH actions workflow changes without adding me in the reviews. Please also add @nicoloboschi as a reviewer to any such PRs.
I have commented on #16539 about reasons why making the change would cause disruption for our CI. Each GitHub API token has a quota of 5000 API calls per hour. Our GitHub token will run out of quota if we start using this workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] maxsxu commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
maxsxu commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1177646770
### Issue 3
Old branches have different workflows from the `master` branch. e.g, [workflows of branch-2.7](https://github.com/apache/pulsar/tree/branch-2.7/.github/workflows)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1182957823
> Compared with other projects' workflows, Pulsar has many workflows and the [list](https://github.com/apache/pulsar/actions) is long.
There's a long history around this. A new workflow to replace multiple workflows was finally merged in master branch in March. Please check the thread https://lists.apache.org/thread/dlsng34zj9q364zk8plw773m12m3std9 and Google doc https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit# for more background.
The [new workflow "pulsar-ci"](https://github.com/apache/pulsar/blob/master/.github/workflows/pulsar-ci.yaml) that reduces the amount of workflows is used in master branch. All maintenance branches continue to use the previous solution of multiple separate workflows. There are some dependencies to the build and migrating current maintenance branches to use "pulsar-ci" workflow would require some effort. I would make sense to migrate at least branch-2.10 to use pulsar-ci. /cc @nicoloboschi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1178415705
I'll try resolving these issues with a new workflow that delete stale workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] maxsxu commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
maxsxu commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1177639734
@Anonymitaet Thanks for bringing this out!
FYI, I made a list of **invalid** (the workflow file does not exist in any branches) and **inactive** (the workflow file exists in an old branch but no workflows runs within a year) workflows about issue 1.
### Invalid Workflows
- https://github.com/apache/pulsar/actions/workflows/ci-build-muiti-os.yaml
- https://github.com/apache/pulsar/actions/workflows/ci-deployment-helm.yaml
- https://github.com/apache/pulsar/actions/workflows/ci-pulsar-build.yml
- https://github.com/apache/pulsar/actions/workflows/ci-workflows.yaml
- https://github.com/apache/pulsar/actions/workflows/cpp-ci.yaml
- https://github.com/apache/pulsar/actions/workflows/ci-integration-ext.yaml
- https://github.com/apache/pulsar/actions/workflows/ci-integration-io.yaml
- https://github.com/apache/pulsar/actions/workflows/ci-integration-java.yaml
- https://github.com/apache/pulsar/actions/workflows/ci-integration-python.yaml
- https://github.com/apache/pulsar/actions/workflows/integration-ci.yaml
- https://github.com/apache/pulsar/actions/workflows/python-whl-osx.yaml
- https://github.com/apache/pulsar/actions/workflows/unit-ci.yaml
### Inactive Workflow
- https://github.com/apache/pulsar/actions/workflows/ci-unit-adaptors.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker-sasl.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker-broker.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker-client.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker-flaky.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker-long-time.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker-others.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker-publish-throttle.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-broker-transaction.yml
- https://github.com/apache/pulsar/actions/workflows/ci-unit-flaky.yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun closed issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
tisonkun closed issue #16439: [improve][workflow] Optimize CI/CD
URL: https://github.com/apache/pulsar/issues/16439
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1182832309
> Abandoned workflows should be hidden and the same workflow should be combined together to make the list looks clearer and tidy.
@Anonymitaet It seems that the priority of this work item should be low since it's only about making the "list look clearer and tidy". Why does it matter? It would be good to have a better rationale for implementing this.
We might be fine with the 400 day retention policy that is now the default in GitHub. There's more context about this in my comment on the PR: https://github.com/apache/pulsar/pull/16539#issuecomment-1182821007 .
The reason why I have been deleting old workflow runs is that it has been causing issues to our CI when there have been so many old workflow runs. there used to be over 550000 archived runs and that might have caused the search index to get corrupted in the GH backend a few times. When the search index got corrupted, the UI would display a warning "We are having problems searching workflow runs. The results may not be complete.". This problem has gone away after contacting GH support (and they rebuilt the search index) and after deleting old runs to reduce the number of runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1309831518
Closed as not actionable.
I believe the case is much better after @lhotari and @nicoloboschi's work on Pulsar CI reworks. Continuous improvements can be made. But holding such an abstract issue saying "generally optimization" is not actionable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dave2wave commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
dave2wave commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1179150058
Here's a thread from April with @lhotari - https://lists.apache.org/thread/n05xp17krb4f3cc0jjkh65ytqk1w6130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16439: [improve][workflow] Optimize CI/CD
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16439:
URL: https://github.com/apache/pulsar/issues/16439#issuecomment-1213633488
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org