You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Lari Hotari <la...@sagire.fi> on 2021/01/28 17:13:54 UTC
[Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Dear Pulsar community members,
Currently, the Pulsar GitHub Actions workflows are consuming the majority
of the shared pool of resources allocated for github.com/apache projects.
Other Apache projects have been impacted and there is a demand to improve
the Pulsar CI
<https://github.com/apache/pulsar/pull/9159#issuecomment-766915396> asap.
In GitHub Actions Runners, the unit of resources is the time that a Runner
is occupied. I observed the workflow runs for handling a single Pull
Request (in my personal fork) and these were the running durations:
Workflow name Duration
CI - Build - MacOS 0:17:23
CI - Go Functions style check 0:02:38
CI - Unit - Brokers - Other 0:15:40
CI - Unit - Brokers - Client Impl 0:16:28
CI - Misc 0:16:51
CI - Unit - Proxy 0:14:23
CI - Go Functions Tests 0:22:08
CI - CPP, Python Tests 0:23:30
CI - Unit 0:42:11
CI - Integration - Sql 1:00:13
CI - Integration - Tiered JCloud 1:00:18
CI - Integration - Tiered FileSystem 1:00:13
CI - Integration - Function State 1:00:12
CI - Integration - Cli 1:10:22
CI - Integration - Transaction 1:16:34
CI - Integration - Process 1:11:23
CI - Shade - Test 1:15:45
CI - Unit - Brokers - Client Api 0:26:13
CI - Unit - Brokers - Broker Group 2 0:35:05
CI - Integration - Standalone 0:45:29
CI - Integration - Messaging 1:00:23
CI - Integration - Thread 1:00:19
CI - Integration - Backwards Compatibility 1:00:19
CI - Integration - Schema 1:00:19
CI - Unit - Brokers - Broker Group 1 2:02:31
TOTAL 19:36:50
*In this case, the total resource consumption of GitHub Actions Runners is
19 hours 36 minutes 50 seconds for a single pull request to apache/pulsar.*
Since GitHub Actions Runner resource pool utilization is very high, this
leads to the build queue to grow and take a long time to process.
I have been looking for ways to improve the Pulsar CI for the last 3
months. During this period I worked on a few experiments. The learnings
from the past experiments are documented at a high level in the following
draft PIP document.
*The draft PIP "Changes to GitHub Actions based Pulsar CI" document is a
Google doc:*
https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
*Please participate* so that we get the plan adjusted based on the feedback
asap. If there's already a similar effort ongoing, I hope we can join
efforts.
*Let's fix Pulsar CI!*
BR, Lari
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <lh...@apache.org>.
I have created a separate PR to make pulsarbot support the new GitHub Actions feature of rerunning failed jobs (instead of all jobs in a workflow). Please review https://github.com/apache/pulsar-test-infra/pull/27 too.
-Lari
On 2022/03/23 10:46:10 Lari Hotari wrote:
> I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
>
> BR,
> -Lari
>
> On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> > Lari,
> >
> > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
> >
> > > I have resumed work to improve our GitHub Actions based Pulsar CI.
> > >
> > > Last year, I worked on a proof-of-concept which significantly reduced the
> > > resource consumption and improved the usability of the build by combining
> > > multiple workflows into a single larger workflow.
> > >
> > > The showstopper a year ago was the lack of being able to re-run a single
> > > failed job in a larger workflow.
> > > GitHub has since then delivered this feature and no showstoppers are
> > > present.
> > >
> > > I have been posting updates to
> > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> > > about the progress.
> > > I have rebased the changes from last year's PoC, and I'm finalizing and
> > > testing the changes in my fork under
> > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> > > apache/pulsar, when the refactoring is ready.
> > >
> >
> > This is great news !
> >
> > Looking forward to your patch
> >
> > Enrico
> >
> >
> >
> > > -Lari
> > >
> > > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > > The prototype has demonstrated about 60% reduction in
> > > > resource consumption.
> > > >
> > > > It is hard to quantify. Merging them into one large workflow can result
> > > in
> > > > more failures. Re-running those failures can consume resources as well.
> > > >
> > > > > Isn't it urgent to resolve it?
> > > >
> > > > I think we are in a stage that gives us breathing room to fix flaky tests
> > > > and solve other problems, no?
> > > > I don't mean we stop the effort here. I mean we have other enhancements
> > > > that we can do to improve the situation.
> > > > Once we get into a position where the flakiness is reduced, we can merge
> > > > them into one workflow.
> > > >
> > > > - Sijie
> > > >
> > > >
> > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> > > >
> > > > > Thanks for the feedback Sijie.
> > > > >
> > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > objection,
> > > > > > please go ahead and not need to wait for approval.
> > > > > > If people raise concerns, please address the concerns.
> > > > >
> > > > > You and Ali have raised concerns about changing the existing GitHub
> > > Actions
> > > > > workflows in a way where multiple workflows would be combined to a
> > > single
> > > > > workflow. Before proceeding, there is a need to address the concerns.
> > > We
> > > > > might end up with a completely different type of solution of what has
> > > been
> > > > > proposed initially. :)
> > > > >
> > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > workflows
> > > > > > into one giant workflow.
> > > > >
> > > > > I agree that addressing flaky tests is favorable. The main reason for
> > > PIP
> > > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > > > 2) Reduce lead times for Pull Request feedback
> > > > > We cannot ignore these problems. If we don't change anything, the
> > > problems
> > > > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > > > resource consumption. Measuring the lead times hasn't been done in the
> > > > > prototype, but since the reason for long lead times has been long build
> > > > > queues due to excessive resource consumption, it's likely that the lead
> > > > > times would be reduced.
> > > > >
> > > > > I know that switching to a single workflow isn't the only solution to
> > > the
> > > > > above problems. I had a discussion with Ali. He recommended reducing
> > > the
> > > > > modules in Pulsar repository (PIP-62), reducing the docker container
> > > size
> > > > > and improving the Pulsar Broker unit test harness so that tests using
> > > it
> > > > > would be less flaky and that it would be easier to fix the issues in
> > > > > failing test when there would be better information about what was the
> > > > > state problem that caused the test to fail.
> > > > >
> > > > > As mentioned in the earlier email about the optimizations in the
> > > Pulsar CI
> > > > > refactoring prototype, the main benefits come from reusing binary
> > > artifacts
> > > > > from previous build stages so that each job doesn't have to build
> > > > > everything from scratch. This becomes irrelevant when the build is very
> > > > > fast and there isn't a benefit of reusing artifacts.
> > > > > This means that it's possible to resolve the resource consumption
> > > problem
> > > > > of Pulsar PR builds in the way that Ali is recommending, without
> > > switching
> > > > > from multiple workflows to a single workflow that can reuse binary
> > > > > artifacts in the build.
> > > > >
> > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > more
> > > > > > on solving flaky tests.
> > > > > > Consider merging them into one workflow when the tests are in a
> > > better
> > > > > > situation.
> > > > >
> > > > > Makes sense for minimizing the risk of change, but we cannot just wait
> > > for
> > > > > things to fix themselves.
> > > > > How long will other Apache projects tolerate the resource consumption
> > > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> > > For
> > > > > example,
> > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > .
> > > > > Isn't it urgent to resolve it?
> > > > >
> > > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> > > CI"
> > > > > based on the community feedback in the upcoming days. That might mean
> > > that
> > > > > the current solution is pivoted. The goal is to solve the problems of
> > > high
> > > > > resource consumption and long lead time for PR build in Pulsar CI.
> > > Please
> > > > > continue to provide feedback so that we get a revisited plan together
> > > that
> > > > > addresses these problems.Thank you!
> > > > >
> > > > > BR,
> > > > > -Lari
> > > > >
> > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > >
> > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > GitHub
> > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > >
> > > > > > I see what you referred to before now. This can be easily done. I
> > > (or any
> > > > > > other committer) can do it for you.
> > > > > >
> > > > > > There is no real blocker for you to continue work even there are
> > > concerns
> > > > > > or it is not listed in the PIP.
> > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > objection,
> > > > > > please go ahead and not need to wait for approval.
> > > > > > If people raise concerns, please address the concerns.
> > > > > >
> > > > > > > The reason why
> > > > > > re-runs happen currently is because of the high flakiness of tests.
> > > > > >
> > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > workflows
> > > > > > into one giant workflow.
> > > > > > It is not about "No pain, no gain". The community has suffered a lot
> > > with
> > > > > > giant workflow before.
> > > > > > There were a lot of committers and contributors working hard to
> > > split one
> > > > > > giant workflow into multiple
> > > > > > current workflows. Unless there is really strong evidence that
> > > merging
> > > > > them
> > > > > > back to one will improve
> > > > > > the entire CI experience, I will still have concerns about one giant
> > > > > > workflow approach.
> > > > > >
> > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > more
> > > > > > on solving flaky tests.
> > > > > > Consider merging them into one workflow when the tests are in a
> > > better
> > > > > > situation.
> > > > > >
> > > > > > > This solution would also require disabling
> > > > > > required status checks
> > > > > >
> > > > > > I don't think it is a good idea to disable status checks. We can
> > > consider
> > > > > > running "dark mode" but it will just overload the action quota.
> > > > > >
> > > > > > Another alternative is to mirror the pull requests into another
> > > Github
> > > > > > account to test that and get more concrete statistics on the
> > > flakiness
> > > > > rate
> > > > > > of one workflow approach.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > >
> > > > > > > Thanks for the feedback, Sijie.
> > > > > > >
> > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > there
> > > > > > > are
> > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > However, I am not sure how much it will help if you run all
> > > workflows
> > > > > > in
> > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > >
> > > > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > > > Addressing flakiness continues to be top-priority. Now that the
> > > Pulsar
> > > > > CI
> > > > > > > workflow prototype is finished, I'll be focusing more in the other
> > > > > draft
> > > > > > > PIP, "Changes to flaky test handling".
> > > > > > > We as a community should address the critical problem that the
> > > current
> > > > > > > retry solution has: it can mask bugs in production code and make
> > > the
> > > > > > build
> > > > > > > pass and allow changes to be merged that cause regressions.
> > > > > > > It's a false sense of security what the green builds after all the
> > > > > > retries
> > > > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > > > addressing this.
> > > > > > >
> > > > > > > If something doesn't work, it can be adapted and improved. Changes
> > > can
> > > > > be
> > > > > > > rolled back and revisited when things go worse. We also need a
> > > leap of
> > > > > > > faith.
> > > > > > > "No pain, no gain", like any change, it will be painful at first,
> > > but
> > > > > we
> > > > > > > will get over the bump.
> > > > > > >
> > > > > > > > Secondly, your test has been done in your folk where there are
> > > not a
> > > > > > lot
> > > > > > > of
> > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > approach
> > > > > > will
> > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > concurrent
> > > > > > > > pull requests in your account to prove your approach doesn't
> > > bring
> > > > > side
> > > > > > > > effects?
> > > > > > >
> > > > > > > One possibility to address this is to introduce the new workflow
> > > in a
> > > > > > mode
> > > > > > > where you need to opt-in to the new workflow in some way.
> > > > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > > > It might be possible to configure the existing workflow and this
> > > new
> > > > > > > workflow in a way where some condition (for example whitelisted
> > > github
> > > > > > user
> > > > > > > name or a certain keyword in the PR title/description) chooses
> > > either
> > > > > one
> > > > > > > for the Pull request. This solution would also require disabling
> > > > > > > required status checks ("Require status checks to pass before
> > > merging"
> > > > > > > feature in GitHub branch protection rules) since the names of the
> > > > > checks
> > > > > > > would be different.
> > > > > > >
> > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > without
> > > > > > > > merging them into one giant workflow?
> > > > > > >
> > > > > > > This is what I have been doing. All individual optimizations have
> > > > > already
> > > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > > > have
> > > > > > > already been merged.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > > There's also 2 build related PRs from yesterday which haven't been
> > > > > merged
> > > > > > > yet:
> > > > > > > Fix Maven download issues (ported from the prototype to our
> > > existing
> > > > > > Pulsar
> > > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > > improve Maven module build order (required for more efficient
> > > builds
> > > > > that
> > > > > > > selectively build required artifacts):
> > > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > > >
> > > > > > > There aren't many optimizations left that could be ported from the
> > > > > > > prototype to the existing workflow. There are a few, but the
> > > impact is
> > > > > > > minor.
> > > > > > > The reason for this is that the optimization with the greatest
> > > impact
> > > > > are
> > > > > > > the ones that build a binary artifacts (maven libs, docker images)
> > > once
> > > > > > and
> > > > > > > share it with the downstream jobs in the pipeline.
> > > > > > > Applying this type of solution has certain challenges when there
> > > are
> > > > > > > multiple separate workflow. Sharing binary artifacts to other
> > > workflows
> > > > > > > would require that the workflow to reuse the artifacts gets
> > > triggered
> > > > > by
> > > > > > > the workflow that produced the artifacts. This wouldn't be secure
> > > or
> > > > > > > practical for handling pull requests. Triggering a workflow
> > > explicitly
> > > > > > > would require a token from the main repository and using that for
> > > pull
> > > > > > > request builds would be a serious security vulnerability. (more
> > > details
> > > > > > > about the GitHub Actions security model in
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > > )
> > > > > > >
> > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > GitHub
> > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> > > wiki
> > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > >
> > > > > > > -Lari
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > This is good progress. However, my main concern is still merging
> > > all
> > > > > > > > workflows into one giant workflow.
> > > > > > > >
> > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > there
> > > > > > > are
> > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > However, I am not sure how much it will help if you run all
> > > workflows
> > > > > > in
> > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > >
> > > > > > > > Secondly, your test has been done in your folk where there are
> > > not a
> > > > > > lot
> > > > > > > of
> > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > approach
> > > > > > will
> > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > concurrent
> > > > > > > > pull requests in your account to prove your approach doesn't
> > > bring
> > > > > side
> > > > > > > > effects?
> > > > > > > >
> > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > without
> > > > > > > > merging them into one giant workflow?
> > > > > > > >
> > > > > > > > - Sijie
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for the feedback Michael.
> > > > > > > > >
> > > > > > > > > > I left a question on the doc about how concurrent runs
> > > affect the
> > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > >
> > > > > > > > > This is a good question. There isn't a clear answer in the
> > > GitHub
> > > > > > > Actions
> > > > > > > > > Cache documentation.
> > > > > > > > >
> > > > > > > > > The documentation is
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > > .
> > > > > > > > > Based on this document and some testing, I have made these
> > > > > > conclusions:
> > > > > > > > > For GitHub Actions Cache, pull requests get executed in the
> > > context
> > > > > > of
> > > > > > > > the
> > > > > > > > > forked repository.
> > > > > > > > > The workflow triggered by a pull request event can only update
> > > it's
> > > > > > own
> > > > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > > > If there's a cache miss, the entry will get written to the
> > > cache of
> > > > > > the
> > > > > > > > > forked repository. If the PR could to write to the upstream
> > > cache,
> > > > > it
> > > > > > > > would
> > > > > > > > > be a security issue since this would be vulnerable to cache
> > > > > poisoning
> > > > > > > > > attacks. Each repository has a 5GB quota for writes. The
> > > entries
> > > > > are
> > > > > > > kept
> > > > > > > > > up to 7 days.
> > > > > > > > > The performance is fairly good. Loading docker images from the
> > > > > > > repository
> > > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> > > (the
> > > > > > > > > performance of the GHA cache is most likely higher since this
> > > is
> > > > > the
> > > > > > > > > throughput for docker load / docker save)
> > > > > > > > >
> > > > > > > > > If a single repository has a lot of concurrent jobs, it could
> > > start
> > > > > > > > > evicting caches.
> > > > > > > > > However that isn't likely to happen with the way Pulsar is
> > > > > developed
> > > > > > > > since
> > > > > > > > > pull requests are created from personal forks.
> > > > > > > > >
> > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > reference
> > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > assuming
> > > > > > > > > that
> > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > completion,
> > > > > > > > and
> > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > I
> > > > > > > haven't
> > > > > > > > > > worked with github actions before.
> > > > > > > > >
> > > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> > > jobs
> > > > > > in
> > > > > > > > the
> > > > > > > > > matrix if one fails. Other parallel flows would run to
> > > completion
> > > > > by
> > > > > > > > > default.
> > > > > > > > > In the prototype, I have added a Github script step to each
> > > job to
> > > > > > > cancel
> > > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > > Here's an example:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > > >
> > > > > > > > > The prototype follows a "fail fast" design. When a failure
> > > occurs,
> > > > > > fail
> > > > > > > > > fast and don't continue with other jobs.
> > > > > > > > > The benefit of this is that it reduces resource consumption.
> > > This
> > > > > > helps
> > > > > > > > > keep the build queue short.
> > > > > > > > > When the build queue is short, developers get quick feedback
> > > from
> > > > > CI.
> > > > > > > > >
> > > > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > > > *I'm hoping to start a separate document on low level details
> > > when
> > > > > > > there
> > > > > > > > is
> > > > > > > > > a high level acceptance of the proposed "Changes to GitHub
> > > Actions
> > > > > > > based
> > > > > > > > > Pulsar CI".*
> > > > > > > > > Together we can make this happen. We need decisions too. This
> > > > > > proposal
> > > > > > > > > cannot stay as a draft forever.
> > > > > > > > > I'm looking forward to hearing from the Pulsar community,
> > > Pulsar
> > > > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > > > >
> > > > > > > > > BR,
> > > > > > > > > -Lari
> > > > > > > > >
> > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > > > mikemarsh17@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > > > overall,
> > > > > > > > it
> > > > > > > > > > looks good to me.
> > > > > > > > > >
> > > > > > > > > > I left a question on the doc about how concurrent runs
> > > affect the
> > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > >
> > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > reference
> > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > assuming
> > > > > > > > > that
> > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > completion,
> > > > > > > > and
> > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > I
> > > > > > > haven't
> > > > > > > > > > worked with github actions before.
> > > > > > > > > >
> > > > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> > > community
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > > lari.hotari@sagire.fi
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > >
> > > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> > > has
> > > > > > gone
> > > > > > > > > > forward
> > > > > > > > > > > > based on your feedback. Here are some updates about the
> > > work.
> > > > > > > > > > > >
> > > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > There's a *detailed status update in the document about a
> > > > > > > prototype
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> > > lot of
> > > > > > > > > > improvements
> > > > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > > > technical
> > > > > > > > > > > > obstacles.
> > > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> > > docker
> > > > > > > > images.
> > > > > > > > > A
> > > > > > > > > > > lot
> > > > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > > > build
> > > > > > to
> > > > > > > > > > enable
> > > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> > > for all
> > > > > > PR
> > > > > > > > > > reviews
> > > > > > > > > > > > and feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> > > Pulsar
> > > > > > CI"
> > > > > > > > work
> > > > > > > > > > has
> > > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > > > build
> > > > > > > and
> > > > > > > > > to
> > > > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > > > productivity*
> > > > > > > > > > when
> > > > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> > > feedback.
> > > > > > The
> > > > > > > > > > > prototype
> > > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > > >
> > > > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > > > resource
> > > > > > > > > > > consumption
> > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > > > observed
> > > > > > > > > when
> > > > > > > > > > > the
> > > > > > > > > > > > work began.
> > > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> > > build,
> > > > > the
> > > > > > > > > resource
> > > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > > *This is about 60% reduction in resource consumption.*
> > > The
> > > > > > whole
> > > > > > > > > > pipeline
> > > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > > >
> > > > > > > > > > > > Here's a breakdown of the duration (resource
> > > consumption) of
> > > > > > each
> > > > > > > > > build
> > > > > > > > > > > job
> > > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > > > 0:06:45
> > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> > > 0:16:08
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> > > 0:37:03
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > > > 0:14:09
> > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > > > 0:29:55
> > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > > > 0:38:32
> > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > > 7:08:57
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > > ).
> > > > > > > > > > > > However, this is not a showstopper since there are ways
> > > to
> > > > > > > address
> > > > > > > > > the
> > > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > > There is a separate PIP for changing the way to handle
> > > flaky
> > > > > > > tests.
> > > > > > > > > You
> > > > > > > > > > > can
> > > > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> > > based
> > > > > > > > Pulsar
> > > > > > > > > > CI"
> > > > > > > > > > > > document's header.
> > > > > > > > > > > >
> > > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > > >
> > > > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > . *It also contains more details of the prototype that
> > > has
> > > > > been
> > > > > > > > > > > > successfully completed.*
> > > > > > > > > > > >
> > > > > > > > > > > > 2) *Please share your feedback and suggest a way
> > > forward.*
> > > > > > > > > > > >
> > > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > > >
> > > > > > > > > > > > BR, Lari
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > > > lari.hotari@sagire.fi
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > > consuming
> > > > > > > the
> > > > > > > > > > > majority
> > > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > > github.com/apache
> > > > > > > > > > > > projects.
> > > > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > > > demand
> > > > > > > to
> > > > > > > > > > > improve
> > > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > > <
> > > > > > > >
> > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > >
> > > > > > > > > > > > asap.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > > > time
> > > > > > > > that a
> > > > > > > > > > > > Runner
> > > > > > > > > > > > > is occupied. I observed the workflow runs for handling
> > > a
> > > > > > single
> > > > > > > > > Pull
> > > > > > > > > > > > > Request (in my personal fork) and these were the
> > > running
> > > > > > > > durations:
> > > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > > >
> > > > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > > > Actions
> > > > > > > > > > Runners
> > > > > > > > > > > > is
> > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > > request to
> > > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > > is
> > > > > very
> > > > > > > > high,
> > > > > > > > > > > this
> > > > > > > > > > > > > leads to the build queue to grow and take a long time
> > > to
> > > > > > > process.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > > for
> > > > > the
> > > > > > > > last
> > > > > > > > > 3
> > > > > > > > > > > > > months. During this period I worked on a few
> > > experiments.
> > > > > The
> > > > > > > > > > learnings
> > > > > > > > > > > > > from the past experiments are documented at a high
> > > level in
> > > > > > the
> > > > > > > > > > > following
> > > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > > >
> > > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > > CI"
> > > > > > > > document
> > > > > > > > > > is
> > > > > > > > > > > a
> > > > > > > > > > > > > Google doc:*
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > > based
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > > feedback asap. If there's already a similar effort
> > > > > ongoing, I
> > > > > > > > hope
> > > > > > > > > we
> > > > > > > > > > > can
> > > > > > > > > > > > > join efforts.
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > > >
> > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Enrico Olivelli <eo...@gmail.com>.
Lari,
I would start a new thread with an 'Heads up'
People may not follow this long thread and miss this piece of information
Great work!
Thank you very much
Enrico
Il Mer 30 Mar 2022, 18:32 Lari Hotari <lh...@apache.org> ha scritto:
> The refactored Pulsar CI workflow PR
> https://github.com/apache/pulsar/pull/14819 has been merged. It unblocks
> CI and makes it work again. Pulsar SQL integration tests are disabled
> temporarily until https://github.com/apache/pulsar/issues/14951 has been
> addressed.
>
> Please rebase your PR or close/reopen it to pick up changes for the new
> Pulsar CI workflow. That is necessary so that PRs can be merged.
>
> -Lari
>
>
> On 2022/03/30 14:08:17 Lari Hotari wrote:
> >
> > Merging the PR is blocked by
> https://github.com/apache/pulsar/issues/14951 .
> >
> > Pulsar SQL doesn't work with Java 11.0.14.1 version. It fails with this
> error message:
> > Exception in thread "main" java.lang.IllegalArgumentException: Cannot
> parse version 11.0.14.1
> > at io.prestosql.server.JavaVersion.parse(JavaVersion.java:76)
> > at
> io.prestosql.server.PrestoSystemRequirements.verifyJavaVersion(PrestoSystemRequirements.java:102)
> > at
> io.prestosql.server.PrestoSystemRequirements.verifyJvmRequirements(PrestoSystemRequirements.java:45)
> > at io.prestosql.server.PrestoServer.run(PrestoServer.java:78)
> > at io.prestosql.$gen.Presto_332____20220330_100314_1.run(Unknown
> Source)
> > at io.prestosql.server.PrestoServer.main(PrestoServer.java:72)
> >
> > I'll apply a workaround to unblock CI.
> >
> > -Lari
> >
> > On 2022/03/30 06:52:38 Lari Hotari wrote:
> > > Thank you for the reviews and feedback. I have started making the
> switch to the new refactored Pulsar CI.
> > >
> > > Merging new PRs are blocked until the switch is ready. The reason for
> this is that I have merged https://github.com/apache/pulsar/pull/14939
> preparing for merging https://github.com/apache/pulsar/pull/14819 .
> > >
> > > The GitHub Actions "required checks" change in the refactored Pulsar
> CI and there can be only one effective set of "required checks" for a
> branch.
> > >
> > > After the new Pulsar CI workflow PR has been merged, each in-progress
> PR has to be closed & immediately reopened to pick up the new workflow and
> the PR build has to run through the new workflow. Another way to pick up
> the new workflow is to rebase the PR (or merge master branch changes to it).
> > >
> > > Please let me know if you experience any issues with the new Pulsar CI
> workflow. I'll be on the #testing channel on Pulsar Slack too.
> > >
> > > -Lari
> > >
> > > On 2022/03/29 15:43:31 Michael Marshall wrote:
> > > > Great work, Lari! It's great news that GitHub's new feature helps
> this
> > > > valuable work move forward. I look forward to seeing your PR merged,
> > > > and I am happy to help resolve any issues that might pop up.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > On Tue, Mar 29, 2022 at 7:55 AM Lari Hotari <lh...@apache.org>
> wrote:
> > > > >
> > > > > The PR has sufficient reviews, and I'll proceed with merging it
> today or tomorrow.
> > > > > Please provide feedback now if you want to do that before the PR
> is merged.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > -Lari
> > > > >
> > > > > On 2022/03/28 20:05:14 Lari Hotari wrote:
> > > > > > The PR https://github.com/apache/pulsar/pull/14819 is now ready
> for review.
> > > > > >
> > > > > > The changes in the PR now use GitHub Actions Artifacts for
> sharing binary files (such as docker images) between the build steps. This
> saves a lot of GitHub Actions VM resources since the docker images are
> built once and shared in downstream jobs.
> > > > > > GitHub Actions Artifacts are meant to be used for sharing data
> between the jobs in a GitHub Actions workflow [1].
> > > > > >
> > > > > > I'm looking forward to your review and feedback on
> https://github.com/apache/pulsar/pull/14819 .
> > > > > >
> > > > > > BR,
> > > > > >
> > > > > > -Lari
> > > > > >
> > > > > > References:
> > > > > > [1] GitHub Actions: Storing workflow data as artifacts -
> https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
> > > > > >
> > > > > > On 2022/03/23 10:46:10 Lari Hotari wrote:
> > > > > > > I have submitted the PR for refactoring the apache/pulsar
> GitHub Actions based CI. Please review
> https://github.com/apache/pulsar/pull/14819 .
> > > > > > >
> > > > > > > BR,
> > > > > > > -Lari
> > > > > > >
> > > > > > > On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> > > > > > > > Lari,
> > > > > > > >
> > > > > > > > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org>
> ha scritto:
> > > > > > > >
> > > > > > > > > I have resumed work to improve our GitHub Actions based
> Pulsar CI.
> > > > > > > > >
> > > > > > > > > Last year, I worked on a proof-of-concept which
> significantly reduced the
> > > > > > > > > resource consumption and improved the usability of the
> build by combining
> > > > > > > > > multiple workflows into a single larger workflow.
> > > > > > > > >
> > > > > > > > > The showstopper a year ago was the lack of being able to
> re-run a single
> > > > > > > > > failed job in a larger workflow.
> > > > > > > > > GitHub has since then delivered this feature and no
> showstoppers are
> > > > > > > > > present.
> > > > > > > > >
> > > > > > > > > I have been posting updates to
> > > > > > > > > https://github.com/apache/pulsar/issues/14401 "Speed up
> CI workflows"
> > > > > > > > > about the progress.
> > > > > > > > > I have rebased the changes from last year's PoC, and I'm
> finalizing and
> > > > > > > > > testing the changes in my fork under
> > > > > > > > > https://github.com/lhotari/pulsar/pull/59 . I'll send a
> PR to
> > > > > > > > > apache/pulsar, when the refactoring is ready.
> > > > > > > > >
> > > > > > > >
> > > > > > > > This is great news !
> > > > > > > >
> > > > > > > > Looking forward to your patch
> > > > > > > >
> > > > > > > > Enrico
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > -Lari
> > > > > > > > >
> > > > > > > > > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > > > > > > > > The prototype has demonstrated about 60% reduction in
> > > > > > > > > > resource consumption.
> > > > > > > > > >
> > > > > > > > > > It is hard to quantify. Merging them into one large
> workflow can result
> > > > > > > > > in
> > > > > > > > > > more failures. Re-running those failures can consume
> resources as well.
> > > > > > > > > >
> > > > > > > > > > > Isn't it urgent to resolve it?
> > > > > > > > > >
> > > > > > > > > > I think we are in a stage that gives us breathing room
> to fix flaky tests
> > > > > > > > > > and solve other problems, no?
> > > > > > > > > > I don't mean we stop the effort here. I mean we have
> other enhancements
> > > > > > > > > > that we can do to improve the situation.
> > > > > > > > > > Once we get into a position where the flakiness is
> reduced, we can merge
> > > > > > > > > > them into one workflow.
> > > > > > > > > >
> > > > > > > > > > - Sijie
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <
> Lari@hotari.net> wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for the feedback Sijie.
> > > > > > > > > > >
> > > > > > > > > > > > We are using a lazy consensus approach. Typically if
> there is no
> > > > > > > > > > > objection,
> > > > > > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > > > > > If people raise concerns, please address the
> concerns.
> > > > > > > > > > >
> > > > > > > > > > > You and Ali have raised concerns about changing the
> existing GitHub
> > > > > > > > > Actions
> > > > > > > > > > > workflows in a way where multiple workflows would be
> combined to a
> > > > > > > > > single
> > > > > > > > > > > workflow. Before proceeding, there is a need to
> address the concerns.
> > > > > > > > > We
> > > > > > > > > > > might end up with a completely different type of
> solution of what has
> > > > > > > > > been
> > > > > > > > > > > proposed initially. :)
> > > > > > > > > > >
> > > > > > > > > > > > Yes. So I am in favor of addressing flaky tests than
> merging all
> > > > > > > > > > > workflows
> > > > > > > > > > > > into one giant workflow.
> > > > > > > > > > >
> > > > > > > > > > > I agree that addressing flaky tests is favorable. The
> main reason for
> > > > > > > > > PIP
> > > > > > > > > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > > > > > > > > 1) Reduce GitHub Action Runner resource consumption of
> Pulsar PR builds
> > > > > > > > > > > 2) Reduce lead times for Pull Request feedback
> > > > > > > > > > > We cannot ignore these problems. If we don't change
> anything, the
> > > > > > > > > problems
> > > > > > > > > > > won't get fixed. The prototype has demonstrated about
> 60% reduction in
> > > > > > > > > > > resource consumption. Measuring the lead times hasn't
> been done in the
> > > > > > > > > > > prototype, but since the reason for long lead times
> has been long build
> > > > > > > > > > > queues due to excessive resource consumption, it's
> likely that the lead
> > > > > > > > > > > times would be reduced.
> > > > > > > > > > >
> > > > > > > > > > > I know that switching to a single workflow isn't the
> only solution to
> > > > > > > > > the
> > > > > > > > > > > above problems. I had a discussion with Ali. He
> recommended reducing
> > > > > > > > > the
> > > > > > > > > > > modules in Pulsar repository (PIP-62), reducing the
> docker container
> > > > > > > > > size
> > > > > > > > > > > and improving the Pulsar Broker unit test harness so
> that tests using
> > > > > > > > > it
> > > > > > > > > > > would be less flaky and that it would be easier to fix
> the issues in
> > > > > > > > > > > failing test when there would be better information
> about what was the
> > > > > > > > > > > state problem that caused the test to fail.
> > > > > > > > > > >
> > > > > > > > > > > As mentioned in the earlier email about the
> optimizations in the
> > > > > > > > > Pulsar CI
> > > > > > > > > > > refactoring prototype, the main benefits come from
> reusing binary
> > > > > > > > > artifacts
> > > > > > > > > > > from previous build stages so that each job doesn't
> have to build
> > > > > > > > > > > everything from scratch. This becomes irrelevant when
> the build is very
> > > > > > > > > > > fast and there isn't a benefit of reusing artifacts.
> > > > > > > > > > > This means that it's possible to resolve the resource
> consumption
> > > > > > > > > problem
> > > > > > > > > > > of Pulsar PR builds in the way that Ali is
> recommending, without
> > > > > > > > > switching
> > > > > > > > > > > from multiple workflows to a single workflow that can
> reuse binary
> > > > > > > > > > > artifacts in the build.
> > > > > > > > > > >
> > > > > > > > > > > > Hence I am +1 to "changes to flaky test handing" and
> suggest focusing
> > > > > > > > > > > more
> > > > > > > > > > > > on solving flaky tests.
> > > > > > > > > > > > Consider merging them into one workflow when the
> tests are in a
> > > > > > > > > better
> > > > > > > > > > > > situation.
> > > > > > > > > > >
> > > > > > > > > > > Makes sense for minimizing the risk of change, but we
> cannot just wait
> > > > > > > > > for
> > > > > > > > > > > things to fix themselves.
> > > > > > > > > > > How long will other Apache projects tolerate the
> resource consumption
> > > > > > > > > > > issues Pulsar is causing in the shared GitHub Actions
> Runner VM quota?
> > > > > > > > > For
> > > > > > > > > > > example,
> > > > > > > > >
> https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > > > .
> > > > > > > > > > > Isn't it urgent to resolve it?
> > > > > > > > > > >
> > > > > > > > > > > I'll revisit the plan for PIP "Changes to GitHub
> Actions based Pulsar
> > > > > > > > > CI"
> > > > > > > > > > > based on the community feedback in the upcoming days.
> That might mean
> > > > > > > > > that
> > > > > > > > > > > the current solution is pivoted. The goal is to solve
> the problems of
> > > > > > > > > high
> > > > > > > > > > > resource consumption and long lead time for PR build
> in Pulsar CI.
> > > > > > > > > Please
> > > > > > > > > > > continue to provide feedback so that we get a
> revisited plan together
> > > > > > > > > that
> > > > > > > > > > > addresses these problems.Thank you!
> > > > > > > > > > >
> > > > > > > > > > > BR,
> > > > > > > > > > > -Lari
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <
> guosijie@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > > *Sijie, how far are we from getting the draft PIP
> "Changes to
> > > > > > > > > GitHub
> > > > > > > > > > > > Actions based Pulsar CI" into an actual PIP that
> gets put on the wiki
> > > > > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > > > > >
> > > > > > > > > > > > I see what you referred to before now. This can be
> easily done. I
> > > > > > > > > (or any
> > > > > > > > > > > > other committer) can do it for you.
> > > > > > > > > > > >
> > > > > > > > > > > > There is no real blocker for you to continue work
> even there are
> > > > > > > > > concerns
> > > > > > > > > > > > or it is not listed in the PIP.
> > > > > > > > > > > > We are using a lazy consensus approach. Typically if
> there is no
> > > > > > > > > > > objection,
> > > > > > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > > > > > If people raise concerns, please address the
> concerns.
> > > > > > > > > > > >
> > > > > > > > > > > > > The reason why
> > > > > > > > > > > > re-runs happen currently is because of the high
> flakiness of tests.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes. So I am in favor of addressing flaky tests than
> merging all
> > > > > > > > > > > workflows
> > > > > > > > > > > > into one giant workflow.
> > > > > > > > > > > > It is not about "No pain, no gain". The community
> has suffered a lot
> > > > > > > > > with
> > > > > > > > > > > > giant workflow before.
> > > > > > > > > > > > There were a lot of committers and contributors
> working hard to
> > > > > > > > > split one
> > > > > > > > > > > > giant workflow into multiple
> > > > > > > > > > > > current workflows. Unless there is really strong
> evidence that
> > > > > > > > > merging
> > > > > > > > > > > them
> > > > > > > > > > > > back to one will improve
> > > > > > > > > > > > the entire CI experience, I will still have concerns
> about one giant
> > > > > > > > > > > > workflow approach.
> > > > > > > > > > > >
> > > > > > > > > > > > Hence I am +1 to "changes to flaky test handing" and
> suggest focusing
> > > > > > > > > > > more
> > > > > > > > > > > > on solving flaky tests.
> > > > > > > > > > > > Consider merging them into one workflow when the
> tests are in a
> > > > > > > > > better
> > > > > > > > > > > > situation.
> > > > > > > > > > > >
> > > > > > > > > > > > > This solution would also require disabling
> > > > > > > > > > > > required status checks
> > > > > > > > > > > >
> > > > > > > > > > > > I don't think it is a good idea to disable status
> checks. We can
> > > > > > > > > consider
> > > > > > > > > > > > running "dark mode" but it will just overload the
> action quota.
> > > > > > > > > > > >
> > > > > > > > > > > > Another alternative is to mirror the pull requests
> into another
> > > > > > > > > Github
> > > > > > > > > > > > account to test that and get more concrete
> statistics on the
> > > > > > > > > flakiness
> > > > > > > > > > > rate
> > > > > > > > > > > > of one workflow approach.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <
> Lari@hotari.net> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the feedback, Sijie.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > The "Fail fast" approach is great. That would be
> super helpful if
> > > > > > > > > > > there
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > multiple workflows and each workflow is
> retryable.
> > > > > > > > > > > > > > However, I am not sure how much it will help if
> you run all
> > > > > > > > > workflows
> > > > > > > > > > > > in
> > > > > > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > > > > >
> > > > > > > > > > > > > We can reduce the need for re-running workflow
> runs. The reason why
> > > > > > > > > > > > > re-runs happen currently is because of high
> flakiness of tests.
> > > > > > > > > > > > > Addressing flakiness continues to be top-priority.
> Now that the
> > > > > > > > > Pulsar
> > > > > > > > > > > CI
> > > > > > > > > > > > > workflow prototype is finished, I'll be focusing
> more in the other
> > > > > > > > > > > draft
> > > > > > > > > > > > > PIP, "Changes to flaky test handling".
> > > > > > > > > > > > > We as a community should address the critical
> problem that the
> > > > > > > > > current
> > > > > > > > > > > > > retry solution has: it can mask bugs in production
> code and make
> > > > > > > > > the
> > > > > > > > > > > > build
> > > > > > > > > > > > > pass and allow changes to be merged that cause
> regressions.
> > > > > > > > > > > > > It's a false sense of security what the green
> builds after all the
> > > > > > > > > > > > retries
> > > > > > > > > > > > > bring us. Bringing Pulsar to the next level in
> stability requires
> > > > > > > > > > > > > addressing this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If something doesn't work, it can be adapted and
> improved. Changes
> > > > > > > > > can
> > > > > > > > > > > be
> > > > > > > > > > > > > rolled back and revisited when things go worse. We
> also need a
> > > > > > > > > leap of
> > > > > > > > > > > > > faith.
> > > > > > > > > > > > > "No pain, no gain", like any change, it will be
> painful at first,
> > > > > > > > > but
> > > > > > > > > > > we
> > > > > > > > > > > > > will get over the bump.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Secondly, your test has been done in your folk
> where there are
> > > > > > > > > not a
> > > > > > > > > > > > lot
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > concurrent pushes and pull requests. I am not
> sure how your
> > > > > > > > > approach
> > > > > > > > > > > > will
> > > > > > > > > > > > > > behave once it is merged into master. Can you
> simulate multiple
> > > > > > > > > > > > > concurrent
> > > > > > > > > > > > > > pull requests in your account to prove your
> approach doesn't
> > > > > > > > > bring
> > > > > > > > > > > side
> > > > > > > > > > > > > > effects?
> > > > > > > > > > > > >
> > > > > > > > > > > > > One possibility to address this is to introduce
> the new workflow
> > > > > > > > > in a
> > > > > > > > > > > > mode
> > > > > > > > > > > > > where you need to opt-in to the new workflow in
> some way.
> > > > > > > > > > > > > This was an idea brought up by my colleagues
> Enrico and Andrey.
> > > > > > > > > > > > > It might be possible to configure the existing
> workflow and this
> > > > > > > > > new
> > > > > > > > > > > > > workflow in a way where some condition (for
> example whitelisted
> > > > > > > > > github
> > > > > > > > > > > > user
> > > > > > > > > > > > > name or a certain keyword in the PR
> title/description) chooses
> > > > > > > > > either
> > > > > > > > > > > one
> > > > > > > > > > > > > for the Pull request. This solution would also
> require disabling
> > > > > > > > > > > > > required status checks ("Require status checks to
> pass before
> > > > > > > > > merging"
> > > > > > > > > > > > > feature in GitHub branch protection rules) since
> the names of the
> > > > > > > > > > > checks
> > > > > > > > > > > > > would be different.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Lastly, can we apply those optimizations to
> current workflows
> > > > > > > > > without
> > > > > > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is what I have been doing. All individual
> optimizations have
> > > > > > > > > > > already
> > > > > > > > > > > > > been sent as PRs in the last months. I guess
> there's 20-30 PRs that
> > > > > > > > > > > have
> > > > > > > > > > > > > already been merged.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > > > > > > > > There's also 2 build related PRs from yesterday
> which haven't been
> > > > > > > > > > > merged
> > > > > > > > > > > > > yet:
> > > > > > > > > > > > > Fix Maven download issues (ported from the
> prototype to our
> > > > > > > > > existing
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > > > > > > > > improve Maven module build order (required for
> more efficient
> > > > > > > > > builds
> > > > > > > > > > > that
> > > > > > > > > > > > > selectively build required artifacts):
> > > > > > > > > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > > > > > > > > >
> > > > > > > > > > > > > There aren't many optimizations left that could be
> ported from the
> > > > > > > > > > > > > prototype to the existing workflow. There are a
> few, but the
> > > > > > > > > impact is
> > > > > > > > > > > > > minor.
> > > > > > > > > > > > > The reason for this is that the optimization with
> the greatest
> > > > > > > > > impact
> > > > > > > > > > > are
> > > > > > > > > > > > > the ones that build a binary artifacts (maven
> libs, docker images)
> > > > > > > > > once
> > > > > > > > > > > > and
> > > > > > > > > > > > > share it with the downstream jobs in the pipeline.
> > > > > > > > > > > > > Applying this type of solution has certain
> challenges when there
> > > > > > > > > are
> > > > > > > > > > > > > multiple separate workflow. Sharing binary
> artifacts to other
> > > > > > > > > workflows
> > > > > > > > > > > > > would require that the workflow to reuse the
> artifacts gets
> > > > > > > > > triggered
> > > > > > > > > > > by
> > > > > > > > > > > > > the workflow that produced the artifacts. This
> wouldn't be secure
> > > > > > > > > or
> > > > > > > > > > > > > practical for handling pull requests. Triggering a
> workflow
> > > > > > > > > explicitly
> > > > > > > > > > > > > would require a token from the main repository and
> using that for
> > > > > > > > > pull
> > > > > > > > > > > > > request builds would be a serious security
> vulnerability. (more
> > > > > > > > > details
> > > > > > > > > > > > > about the GitHub Actions security model in
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > > > > > > > > )
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Sijie, how far are we from getting the draft PIP
> "Changes to
> > > > > > > > > GitHub
> > > > > > > > > > > > > Actions based Pulsar CI" into an actual PIP that
> gets put on the
> > > > > > > > > wiki
> > > > > > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Lari
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <
> guosijie@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > This is good progress. However, my main concern
> is still merging
> > > > > > > > > all
> > > > > > > > > > > > > > workflows into one giant workflow.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The "Fail fast" approach is great. That would be
> super helpful if
> > > > > > > > > > > there
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > multiple workflows and each workflow is
> retryable.
> > > > > > > > > > > > > > However, I am not sure how much it will help if
> you run all
> > > > > > > > > workflows
> > > > > > > > > > > > in
> > > > > > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Secondly, your test has been done in your folk
> where there are
> > > > > > > > > not a
> > > > > > > > > > > > lot
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > concurrent pushes and pull requests. I am not
> sure how your
> > > > > > > > > approach
> > > > > > > > > > > > will
> > > > > > > > > > > > > > behave once it is merged into master. Can you
> simulate multiple
> > > > > > > > > > > > > concurrent
> > > > > > > > > > > > > > pull requests in your account to prove your
> approach doesn't
> > > > > > > > > bring
> > > > > > > > > > > side
> > > > > > > > > > > > > > effects?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Lastly, can we apply those optimizations to
> current workflows
> > > > > > > > > without
> > > > > > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - Sijie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <
> Lari@hotari.net>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the feedback Michael.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I left a question on the doc about how
> concurrent runs
> > > > > > > > > affect the
> > > > > > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is a good question. There isn't a clear
> answer in the
> > > > > > > > > GitHub
> > > > > > > > > > > > > Actions
> > > > > > > > > > > > > > > Cache documentation.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The documentation is
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > Based on this document and some testing, I
> have made these
> > > > > > > > > > > > conclusions:
> > > > > > > > > > > > > > > For GitHub Actions Cache, pull requests get
> executed in the
> > > > > > > > > context
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > forked repository.
> > > > > > > > > > > > > > > The workflow triggered by a pull request event
> can only update
> > > > > > > > > it's
> > > > > > > > > > > > own
> > > > > > > > > > > > > > > cache. It has read-only access to upstream
> caches.
> > > > > > > > > > > > > > > If there's a cache miss, the entry will get
> written to the
> > > > > > > > > cache of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > forked repository. If the PR could to write to
> the upstream
> > > > > > > > > cache,
> > > > > > > > > > > it
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > be a security issue since this would be
> vulnerable to cache
> > > > > > > > > > > poisoning
> > > > > > > > > > > > > > > attacks. Each repository has a 5GB quota for
> writes. The
> > > > > > > > > entries
> > > > > > > > > > > are
> > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > up to 7 days.
> > > > > > > > > > > > > > > The performance is fairly good. Loading docker
> images from the
> > > > > > > > > > > > > repository
> > > > > > > > > > > > > > > happens about 15MB/s. Writing is 2-3x slower,
> about 5-7MB/s.
> > > > > > > > > (the
> > > > > > > > > > > > > > > performance of the GHA cache is most likely
> higher since this
> > > > > > > > > is
> > > > > > > > > > > the
> > > > > > > > > > > > > > > throughput for docker load / docker save)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If a single repository has a lot of concurrent
> jobs, it could
> > > > > > > > > start
> > > > > > > > > > > > > > > evicting caches.
> > > > > > > > > > > > > > > However that isn't likely to happen with the
> way Pulsar is
> > > > > > > > > > > developed
> > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > pull requests are created from personal forks.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I also think it could be helpful to
> explicitly document, or
> > > > > > > > > > > > reference
> > > > > > > > > > > > > > > > github documentation, on how failure will
> affect the DAG. I'm
> > > > > > > > > > > > > assuming
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > if an action fails, its parallel peer
> actions will run to
> > > > > > > > > > > > completion,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > that the rest of the remaining stages will
> get canceled, but
> > > > > > > > > I
> > > > > > > > > > > > > haven't
> > > > > > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For matrix jobs, "fail fast" is the default,
> which cancels all
> > > > > > > > > jobs
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > matrix if one fails. Other parallel flows
> would run to
> > > > > > > > > completion
> > > > > > > > > > > by
> > > > > > > > > > > > > > > default.
> > > > > > > > > > > > > > > In the prototype, I have added a Github script
> step to each
> > > > > > > > > job to
> > > > > > > > > > > > > cancel
> > > > > > > > > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The prototype follows a "fail fast" design.
> When a failure
> > > > > > > > > occurs,
> > > > > > > > > > > > fail
> > > > > > > > > > > > > > > fast and don't continue with other jobs.
> > > > > > > > > > > > > > > The benefit of this is that it reduces
> resource consumption.
> > > > > > > > > This
> > > > > > > > > > > > helps
> > > > > > > > > > > > > > > keep the build queue short.
> > > > > > > > > > > > > > > When the build queue is short, developers get
> quick feedback
> > > > > > > > > from
> > > > > > > > > > > CI.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Documenting all details in the PIP document
> isn't practical.
> > > > > > > > > > > > > > > *I'm hoping to start a separate document on
> low level details
> > > > > > > > > when
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a high level acceptance of the proposed
> "Changes to GitHub
> > > > > > > > > Actions
> > > > > > > > > > > > > based
> > > > > > > > > > > > > > > Pulsar CI".*
> > > > > > > > > > > > > > > Together we can make this happen. We need
> decisions too. This
> > > > > > > > > > > > proposal
> > > > > > > > > > > > > > > cannot stay as a draft forever.
> > > > > > > > > > > > > > > I'm looking forward to hearing from the Pulsar
> community,
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > > committer and Pulsar PMC members how to take
> this forward.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > BR,
> > > > > > > > > > > > > > > -Lari
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael
> Marshall <
> > > > > > > > > > > > > mikemarsh17@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This will be a great improvement. I read
> through the PIP, and
> > > > > > > > > > > > > overall,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > looks good to me.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I left a question on the doc about how
> concurrent runs
> > > > > > > > > affect the
> > > > > > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I also think it could be helpful to
> explicitly document, or
> > > > > > > > > > > > reference
> > > > > > > > > > > > > > > > github documentation, on how failure will
> affect the DAG. I'm
> > > > > > > > > > > > > assuming
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > if an action fails, its parallel peer
> actions will run to
> > > > > > > > > > > > completion,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > that the rest of the remaining stages will
> get canceled, but
> > > > > > > > > I
> > > > > > > > > > > > > haven't
> > > > > > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for all of the work you've put in so
> far.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <
> uvaraj6@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This is great news. Thanks Hari , Mateo
> and pulsar
> > > > > > > > > community
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The work on "Changes to GitHub Actions
> based Pulsar CI"
> > > > > > > > > has
> > > > > > > > > > > > gone
> > > > > > > > > > > > > > > > forward
> > > > > > > > > > > > > > > > > > based on your feedback. Here are some
> updates about the
> > > > > > > > > work.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > > > > > There's a *detailed status update in the
> document about a
> > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > refactored Pulsar CI GitHub Actions
> based workflow*.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for all the suggestions and
> feedback by now. A
> > > > > > > > > lot of
> > > > > > > > > > > > > > > > improvements
> > > > > > > > > > > > > > > > > > have been made by the Pulsar
> contributors to overcome the
> > > > > > > > > > > > > technical
> > > > > > > > > > > > > > > > > > obstacles.
> > > > > > > > > > > > > > > > > > Special thanks go to Matteo for reducing
> the sizes of
> > > > > > > > > docker
> > > > > > > > > > > > > > images.
> > > > > > > > > > > > > > > A
> > > > > > > > > > > > > > > > > lot
> > > > > > > > > > > > > > > > > > of small improvements have been made to
> the Pulsar maven
> > > > > > > > > > > build
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > enable
> > > > > > > > > > > > > > > > > > the new refactored GitHub Actions
> workflow. Thank you
> > > > > > > > > for all
> > > > > > > > > > > > PR
> > > > > > > > > > > > > > > > reviews
> > > > > > > > > > > > > > > > > > and feedback.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The main goal of the "Changes to GitHub
> Actions based
> > > > > > > > > Pulsar
> > > > > > > > > > > > CI"
> > > > > > > > > > > > > > work
> > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > been to *reduce the resource consumption
> of the Pulsar CI
> > > > > > > > > > > build
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > speed up Pulsar development by improving
> the developer
> > > > > > > > > > > > > > productivity*
> > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > less time is wasted in waiting for
> Pulsar CI build
> > > > > > > > > feedback.
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As you can see from the email from Jan
> 28 below, *the
> > > > > > > > > > > resource
> > > > > > > > > > > > > > > > > consumption
> > > > > > > > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull
> request that was
> > > > > > > > > > > > > observed
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > work began.
> > > > > > > > > > > > > > > > > > Now, with the prototype of the
> refactored Pulsar CI
> > > > > > > > > build,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > resource
> > > > > > > > > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > > > > > > > > *This is about 60% reduction in resource
> consumption.*
> > > > > > > > > The
> > > > > > > > > > > > whole
> > > > > > > > > > > > > > > > pipeline
> > > > > > > > > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Here's a breakdown of the duration
> (resource
> > > > > > > > > consumption) of
> > > > > > > > > > > > each
> > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > > job
> > > > > > > > > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > > > > > > > > Pulsar CI Build and License check 346
> 0:05:46
> > > > > > > > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python
> clients 683 0:11:23
> > > > > > > > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image
> docker image 405
> > > > > > > > > > > 0:06:45
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker
> Group 1 968
> > > > > > > > > 0:16:08
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker
> Group 2 2223
> > > > > > > > > 0:37:03
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client
> Api 1652 0:27:32
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client
> Impl 916 0:15:16
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other
> 522 0:08:42
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343
> 0:39:03
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414
> 0:06:54
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Backwards
> Compatibility 849
> > > > > > > > > > > > 0:14:09
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490
> 0:24:50
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Messaging
> 857 0:14:17
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468
> 0:07:48
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Standalone
> 286 0:04:46
> > > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Transaction
> 362 0:06:02
> > > > > > > > > > > > > > > > > > Pulsar CI CI - System - Function State
> 699 0:11:39
> > > > > > > > > > > > > > > > > > Pulsar CI CI - System - Tiered
> FileSystem 779 0:12:59
> > > > > > > > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud
> 529 0:08:49
> > > > > > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar
> Connectors - Thread 1795
> > > > > > > > > > > 0:29:55
> > > > > > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar
> Connectors - Process 2312
> > > > > > > > > > > > 0:38:32
> > > > > > > > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > > > > > > > > 7:08:57
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > GitHub Actions doesn't support
> restarting a single job (
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > However, this is not a showstopper since
> there are ways
> > > > > > > > > to
> > > > > > > > > > > > > address
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > > > > > > > > There is a separate PIP for changing the
> way to handle
> > > > > > > > > flaky
> > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > You
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > find the link to that in the "Changes to
> GitHub Actions
> > > > > > > > > based
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > CI"
> > > > > > > > > > > > > > > > > > document's header.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1) *Please take a look at the updated
> PIP document*:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > > > > > . *It also contains more details of the
> prototype that
> > > > > > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > successfully completed.*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2) *Please share your feedback and
> suggest a way
> > > > > > > > > forward.*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari
> Hotari <
> > > > > > > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Currently, the Pulsar GitHub Actions
> workflows are
> > > > > > > > > > > consuming
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > majority
> > > > > > > > > > > > > > > > > > > of the shared pool of resources
> allocated for
> > > > > > > > > > > > > github.com/apache
> > > > > > > > > > > > > > > > > > projects.
> > > > > > > > > > > > > > > > > > > Other Apache projects have been
> impacted and there is a
> > > > > > > > > > > > demand
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > improve
> > > > > > > > > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > >
> > > > > > > > >
> https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > asap.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > In GitHub Actions Runners, the unit of
> resources is the
> > > > > > > > > > > time
> > > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > > Runner
> > > > > > > > > > > > > > > > > > > is occupied. I observed the workflow
> runs for handling
> > > > > > > > > a
> > > > > > > > > > > > single
> > > > > > > > > > > > > > > Pull
> > > > > > > > > > > > > > > > > > > Request (in my personal fork) and
> these were the
> > > > > > > > > running
> > > > > > > > > > > > > > durations:
> > > > > > > > > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Client Impl
> 0:16:28
> > > > > > > > > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > > > > > > > > CI - Integration - Tiered JCloud
> 1:00:18
> > > > > > > > > > > > > > > > > > > CI - Integration - Tiered FileSystem
> 1:00:13
> > > > > > > > > > > > > > > > > > > CI - Integration - Function State
> 1:00:12
> > > > > > > > > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Client Api
> 0:26:13
> > > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2
> 0:35:05
> > > > > > > > > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > > > > > > > > CI - Integration - Backwards
> Compatibility 1:00:19
> > > > > > > > > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1
> 2:02:31
> > > > > > > > > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *In this case, the total resource
> consumption of GitHub
> > > > > > > > > > > > Actions
> > > > > > > > > > > > > > > > Runners
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a
> single pull
> > > > > > > > > request to
> > > > > > > > > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Since GitHub Actions Runner resource
> pool utilization
> > > > > > > > > is
> > > > > > > > > > > very
> > > > > > > > > > > > > > high,
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > leads to the build queue to grow and
> take a long time
> > > > > > > > > to
> > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have been looking for ways to
> improve the Pulsar CI
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > 3
> > > > > > > > > > > > > > > > > > > months. During this period I worked on
> a few
> > > > > > > > > experiments.
> > > > > > > > > > > The
> > > > > > > > > > > > > > > > learnings
> > > > > > > > > > > > > > > > > > > from the past experiments are
> documented at a high
> > > > > > > > > level in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *The draft PIP "Changes to GitHub
> Actions based Pulsar
> > > > > > > > > CI"
> > > > > > > > > > > > > > document
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > Google doc:*
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *Please participate* so that we get
> the plan adjusted
> > > > > > > > > based
> > > > > > > > > > > > on
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > feedback asap. If there's already a
> similar effort
> > > > > > > > > > > ongoing, I
> > > > > > > > > > > > > > hope
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > join efforts.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <lh...@apache.org>.
The refactored Pulsar CI workflow PR https://github.com/apache/pulsar/pull/14819 has been merged. It unblocks CI and makes it work again. Pulsar SQL integration tests are disabled temporarily until https://github.com/apache/pulsar/issues/14951 has been addressed.
Please rebase your PR or close/reopen it to pick up changes for the new Pulsar CI workflow. That is necessary so that PRs can be merged.
-Lari
On 2022/03/30 14:08:17 Lari Hotari wrote:
>
> Merging the PR is blocked by https://github.com/apache/pulsar/issues/14951 .
>
> Pulsar SQL doesn't work with Java 11.0.14.1 version. It fails with this error message:
> Exception in thread "main" java.lang.IllegalArgumentException: Cannot parse version 11.0.14.1
> at io.prestosql.server.JavaVersion.parse(JavaVersion.java:76)
> at io.prestosql.server.PrestoSystemRequirements.verifyJavaVersion(PrestoSystemRequirements.java:102)
> at io.prestosql.server.PrestoSystemRequirements.verifyJvmRequirements(PrestoSystemRequirements.java:45)
> at io.prestosql.server.PrestoServer.run(PrestoServer.java:78)
> at io.prestosql.$gen.Presto_332____20220330_100314_1.run(Unknown Source)
> at io.prestosql.server.PrestoServer.main(PrestoServer.java:72)
>
> I'll apply a workaround to unblock CI.
>
> -Lari
>
> On 2022/03/30 06:52:38 Lari Hotari wrote:
> > Thank you for the reviews and feedback. I have started making the switch to the new refactored Pulsar CI.
> >
> > Merging new PRs are blocked until the switch is ready. The reason for this is that I have merged https://github.com/apache/pulsar/pull/14939 preparing for merging https://github.com/apache/pulsar/pull/14819 .
> >
> > The GitHub Actions "required checks" change in the refactored Pulsar CI and there can be only one effective set of "required checks" for a branch.
> >
> > After the new Pulsar CI workflow PR has been merged, each in-progress PR has to be closed & immediately reopened to pick up the new workflow and the PR build has to run through the new workflow. Another way to pick up the new workflow is to rebase the PR (or merge master branch changes to it).
> >
> > Please let me know if you experience any issues with the new Pulsar CI workflow. I'll be on the #testing channel on Pulsar Slack too.
> >
> > -Lari
> >
> > On 2022/03/29 15:43:31 Michael Marshall wrote:
> > > Great work, Lari! It's great news that GitHub's new feature helps this
> > > valuable work move forward. I look forward to seeing your PR merged,
> > > and I am happy to help resolve any issues that might pop up.
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Tue, Mar 29, 2022 at 7:55 AM Lari Hotari <lh...@apache.org> wrote:
> > > >
> > > > The PR has sufficient reviews, and I'll proceed with merging it today or tomorrow.
> > > > Please provide feedback now if you want to do that before the PR is merged.
> > > >
> > > > Thanks!
> > > >
> > > > -Lari
> > > >
> > > > On 2022/03/28 20:05:14 Lari Hotari wrote:
> > > > > The PR https://github.com/apache/pulsar/pull/14819 is now ready for review.
> > > > >
> > > > > The changes in the PR now use GitHub Actions Artifacts for sharing binary files (such as docker images) between the build steps. This saves a lot of GitHub Actions VM resources since the docker images are built once and shared in downstream jobs.
> > > > > GitHub Actions Artifacts are meant to be used for sharing data between the jobs in a GitHub Actions workflow [1].
> > > > >
> > > > > I'm looking forward to your review and feedback on https://github.com/apache/pulsar/pull/14819 .
> > > > >
> > > > > BR,
> > > > >
> > > > > -Lari
> > > > >
> > > > > References:
> > > > > [1] GitHub Actions: Storing workflow data as artifacts - https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
> > > > >
> > > > > On 2022/03/23 10:46:10 Lari Hotari wrote:
> > > > > > I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
> > > > > >
> > > > > > BR,
> > > > > > -Lari
> > > > > >
> > > > > > On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> > > > > > > Lari,
> > > > > > >
> > > > > > > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
> > > > > > >
> > > > > > > > I have resumed work to improve our GitHub Actions based Pulsar CI.
> > > > > > > >
> > > > > > > > Last year, I worked on a proof-of-concept which significantly reduced the
> > > > > > > > resource consumption and improved the usability of the build by combining
> > > > > > > > multiple workflows into a single larger workflow.
> > > > > > > >
> > > > > > > > The showstopper a year ago was the lack of being able to re-run a single
> > > > > > > > failed job in a larger workflow.
> > > > > > > > GitHub has since then delivered this feature and no showstoppers are
> > > > > > > > present.
> > > > > > > >
> > > > > > > > I have been posting updates to
> > > > > > > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> > > > > > > > about the progress.
> > > > > > > > I have rebased the changes from last year's PoC, and I'm finalizing and
> > > > > > > > testing the changes in my fork under
> > > > > > > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> > > > > > > > apache/pulsar, when the refactoring is ready.
> > > > > > > >
> > > > > > >
> > > > > > > This is great news !
> > > > > > >
> > > > > > > Looking forward to your patch
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > -Lari
> > > > > > > >
> > > > > > > > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > > > > > > > The prototype has demonstrated about 60% reduction in
> > > > > > > > > resource consumption.
> > > > > > > > >
> > > > > > > > > It is hard to quantify. Merging them into one large workflow can result
> > > > > > > > in
> > > > > > > > > more failures. Re-running those failures can consume resources as well.
> > > > > > > > >
> > > > > > > > > > Isn't it urgent to resolve it?
> > > > > > > > >
> > > > > > > > > I think we are in a stage that gives us breathing room to fix flaky tests
> > > > > > > > > and solve other problems, no?
> > > > > > > > > I don't mean we stop the effort here. I mean we have other enhancements
> > > > > > > > > that we can do to improve the situation.
> > > > > > > > > Once we get into a position where the flakiness is reduced, we can merge
> > > > > > > > > them into one workflow.
> > > > > > > > >
> > > > > > > > > - Sijie
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for the feedback Sijie.
> > > > > > > > > >
> > > > > > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > > > > > objection,
> > > > > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > > > > If people raise concerns, please address the concerns.
> > > > > > > > > >
> > > > > > > > > > You and Ali have raised concerns about changing the existing GitHub
> > > > > > > > Actions
> > > > > > > > > > workflows in a way where multiple workflows would be combined to a
> > > > > > > > single
> > > > > > > > > > workflow. Before proceeding, there is a need to address the concerns.
> > > > > > > > We
> > > > > > > > > > might end up with a completely different type of solution of what has
> > > > > > > > been
> > > > > > > > > > proposed initially. :)
> > > > > > > > > >
> > > > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > > > > > workflows
> > > > > > > > > > > into one giant workflow.
> > > > > > > > > >
> > > > > > > > > > I agree that addressing flaky tests is favorable. The main reason for
> > > > > > > > PIP
> > > > > > > > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > > > > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > > > > > > > > 2) Reduce lead times for Pull Request feedback
> > > > > > > > > > We cannot ignore these problems. If we don't change anything, the
> > > > > > > > problems
> > > > > > > > > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > > > > > > > > resource consumption. Measuring the lead times hasn't been done in the
> > > > > > > > > > prototype, but since the reason for long lead times has been long build
> > > > > > > > > > queues due to excessive resource consumption, it's likely that the lead
> > > > > > > > > > times would be reduced.
> > > > > > > > > >
> > > > > > > > > > I know that switching to a single workflow isn't the only solution to
> > > > > > > > the
> > > > > > > > > > above problems. I had a discussion with Ali. He recommended reducing
> > > > > > > > the
> > > > > > > > > > modules in Pulsar repository (PIP-62), reducing the docker container
> > > > > > > > size
> > > > > > > > > > and improving the Pulsar Broker unit test harness so that tests using
> > > > > > > > it
> > > > > > > > > > would be less flaky and that it would be easier to fix the issues in
> > > > > > > > > > failing test when there would be better information about what was the
> > > > > > > > > > state problem that caused the test to fail.
> > > > > > > > > >
> > > > > > > > > > As mentioned in the earlier email about the optimizations in the
> > > > > > > > Pulsar CI
> > > > > > > > > > refactoring prototype, the main benefits come from reusing binary
> > > > > > > > artifacts
> > > > > > > > > > from previous build stages so that each job doesn't have to build
> > > > > > > > > > everything from scratch. This becomes irrelevant when the build is very
> > > > > > > > > > fast and there isn't a benefit of reusing artifacts.
> > > > > > > > > > This means that it's possible to resolve the resource consumption
> > > > > > > > problem
> > > > > > > > > > of Pulsar PR builds in the way that Ali is recommending, without
> > > > > > > > switching
> > > > > > > > > > from multiple workflows to a single workflow that can reuse binary
> > > > > > > > > > artifacts in the build.
> > > > > > > > > >
> > > > > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > > > > > more
> > > > > > > > > > > on solving flaky tests.
> > > > > > > > > > > Consider merging them into one workflow when the tests are in a
> > > > > > > > better
> > > > > > > > > > > situation.
> > > > > > > > > >
> > > > > > > > > > Makes sense for minimizing the risk of change, but we cannot just wait
> > > > > > > > for
> > > > > > > > > > things to fix themselves.
> > > > > > > > > > How long will other Apache projects tolerate the resource consumption
> > > > > > > > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> > > > > > > > For
> > > > > > > > > > example,
> > > > > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > > .
> > > > > > > > > > Isn't it urgent to resolve it?
> > > > > > > > > >
> > > > > > > > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> > > > > > > > CI"
> > > > > > > > > > based on the community feedback in the upcoming days. That might mean
> > > > > > > > that
> > > > > > > > > > the current solution is pivoted. The goal is to solve the problems of
> > > > > > > > high
> > > > > > > > > > resource consumption and long lead time for PR build in Pulsar CI.
> > > > > > > > Please
> > > > > > > > > > continue to provide feedback so that we get a revisited plan together
> > > > > > > > that
> > > > > > > > > > addresses these problems.Thank you!
> > > > > > > > > >
> > > > > > > > > > BR,
> > > > > > > > > > -Lari
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > > > > > GitHub
> > > > > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > > > >
> > > > > > > > > > > I see what you referred to before now. This can be easily done. I
> > > > > > > > (or any
> > > > > > > > > > > other committer) can do it for you.
> > > > > > > > > > >
> > > > > > > > > > > There is no real blocker for you to continue work even there are
> > > > > > > > concerns
> > > > > > > > > > > or it is not listed in the PIP.
> > > > > > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > > > > > objection,
> > > > > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > > > > If people raise concerns, please address the concerns.
> > > > > > > > > > >
> > > > > > > > > > > > The reason why
> > > > > > > > > > > re-runs happen currently is because of the high flakiness of tests.
> > > > > > > > > > >
> > > > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > > > > > workflows
> > > > > > > > > > > into one giant workflow.
> > > > > > > > > > > It is not about "No pain, no gain". The community has suffered a lot
> > > > > > > > with
> > > > > > > > > > > giant workflow before.
> > > > > > > > > > > There were a lot of committers and contributors working hard to
> > > > > > > > split one
> > > > > > > > > > > giant workflow into multiple
> > > > > > > > > > > current workflows. Unless there is really strong evidence that
> > > > > > > > merging
> > > > > > > > > > them
> > > > > > > > > > > back to one will improve
> > > > > > > > > > > the entire CI experience, I will still have concerns about one giant
> > > > > > > > > > > workflow approach.
> > > > > > > > > > >
> > > > > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > > > > > more
> > > > > > > > > > > on solving flaky tests.
> > > > > > > > > > > Consider merging them into one workflow when the tests are in a
> > > > > > > > better
> > > > > > > > > > > situation.
> > > > > > > > > > >
> > > > > > > > > > > > This solution would also require disabling
> > > > > > > > > > > required status checks
> > > > > > > > > > >
> > > > > > > > > > > I don't think it is a good idea to disable status checks. We can
> > > > > > > > consider
> > > > > > > > > > > running "dark mode" but it will just overload the action quota.
> > > > > > > > > > >
> > > > > > > > > > > Another alternative is to mirror the pull requests into another
> > > > > > > > Github
> > > > > > > > > > > account to test that and get more concrete statistics on the
> > > > > > > > flakiness
> > > > > > > > > > rate
> > > > > > > > > > > of one workflow approach.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for the feedback, Sijie.
> > > > > > > > > > > >
> > > > > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > > > > > there
> > > > > > > > > > > > are
> > > > > > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > > > > > However, I am not sure how much it will help if you run all
> > > > > > > > workflows
> > > > > > > > > > > in
> > > > > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > > > >
> > > > > > > > > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > > > > > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > > > > > > > > Addressing flakiness continues to be top-priority. Now that the
> > > > > > > > Pulsar
> > > > > > > > > > CI
> > > > > > > > > > > > workflow prototype is finished, I'll be focusing more in the other
> > > > > > > > > > draft
> > > > > > > > > > > > PIP, "Changes to flaky test handling".
> > > > > > > > > > > > We as a community should address the critical problem that the
> > > > > > > > current
> > > > > > > > > > > > retry solution has: it can mask bugs in production code and make
> > > > > > > > the
> > > > > > > > > > > build
> > > > > > > > > > > > pass and allow changes to be merged that cause regressions.
> > > > > > > > > > > > It's a false sense of security what the green builds after all the
> > > > > > > > > > > retries
> > > > > > > > > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > > > > > > > > addressing this.
> > > > > > > > > > > >
> > > > > > > > > > > > If something doesn't work, it can be adapted and improved. Changes
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > rolled back and revisited when things go worse. We also need a
> > > > > > > > leap of
> > > > > > > > > > > > faith.
> > > > > > > > > > > > "No pain, no gain", like any change, it will be painful at first,
> > > > > > > > but
> > > > > > > > > > we
> > > > > > > > > > > > will get over the bump.
> > > > > > > > > > > >
> > > > > > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > > > > > not a
> > > > > > > > > > > lot
> > > > > > > > > > > > of
> > > > > > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > > > > > approach
> > > > > > > > > > > will
> > > > > > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > > > > > concurrent
> > > > > > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > > > > > bring
> > > > > > > > > > side
> > > > > > > > > > > > > effects?
> > > > > > > > > > > >
> > > > > > > > > > > > One possibility to address this is to introduce the new workflow
> > > > > > > > in a
> > > > > > > > > > > mode
> > > > > > > > > > > > where you need to opt-in to the new workflow in some way.
> > > > > > > > > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > > > > > > > > It might be possible to configure the existing workflow and this
> > > > > > > > new
> > > > > > > > > > > > workflow in a way where some condition (for example whitelisted
> > > > > > > > github
> > > > > > > > > > > user
> > > > > > > > > > > > name or a certain keyword in the PR title/description) chooses
> > > > > > > > either
> > > > > > > > > > one
> > > > > > > > > > > > for the Pull request. This solution would also require disabling
> > > > > > > > > > > > required status checks ("Require status checks to pass before
> > > > > > > > merging"
> > > > > > > > > > > > feature in GitHub branch protection rules) since the names of the
> > > > > > > > > > checks
> > > > > > > > > > > > would be different.
> > > > > > > > > > > >
> > > > > > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > > > > > without
> > > > > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > > > >
> > > > > > > > > > > > This is what I have been doing. All individual optimizations have
> > > > > > > > > > already
> > > > > > > > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > > > > > > > > have
> > > > > > > > > > > > already been merged.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > > > > > > > There's also 2 build related PRs from yesterday which haven't been
> > > > > > > > > > merged
> > > > > > > > > > > > yet:
> > > > > > > > > > > > Fix Maven download issues (ported from the prototype to our
> > > > > > > > existing
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > > > > > > > improve Maven module build order (required for more efficient
> > > > > > > > builds
> > > > > > > > > > that
> > > > > > > > > > > > selectively build required artifacts):
> > > > > > > > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > > > > > > > >
> > > > > > > > > > > > There aren't many optimizations left that could be ported from the
> > > > > > > > > > > > prototype to the existing workflow. There are a few, but the
> > > > > > > > impact is
> > > > > > > > > > > > minor.
> > > > > > > > > > > > The reason for this is that the optimization with the greatest
> > > > > > > > impact
> > > > > > > > > > are
> > > > > > > > > > > > the ones that build a binary artifacts (maven libs, docker images)
> > > > > > > > once
> > > > > > > > > > > and
> > > > > > > > > > > > share it with the downstream jobs in the pipeline.
> > > > > > > > > > > > Applying this type of solution has certain challenges when there
> > > > > > > > are
> > > > > > > > > > > > multiple separate workflow. Sharing binary artifacts to other
> > > > > > > > workflows
> > > > > > > > > > > > would require that the workflow to reuse the artifacts gets
> > > > > > > > triggered
> > > > > > > > > > by
> > > > > > > > > > > > the workflow that produced the artifacts. This wouldn't be secure
> > > > > > > > or
> > > > > > > > > > > > practical for handling pull requests. Triggering a workflow
> > > > > > > > explicitly
> > > > > > > > > > > > would require a token from the main repository and using that for
> > > > > > > > pull
> > > > > > > > > > > > request builds would be a serious security vulnerability. (more
> > > > > > > > details
> > > > > > > > > > > > about the GitHub Actions security model in
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > > > > > > > )
> > > > > > > > > > > >
> > > > > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > > > > > GitHub
> > > > > > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> > > > > > > > wiki
> > > > > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > > > > >
> > > > > > > > > > > > -Lari
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > This is good progress. However, my main concern is still merging
> > > > > > > > all
> > > > > > > > > > > > > workflows into one giant workflow.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > > > > > there
> > > > > > > > > > > > are
> > > > > > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > > > > > However, I am not sure how much it will help if you run all
> > > > > > > > workflows
> > > > > > > > > > > in
> > > > > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > > > > > not a
> > > > > > > > > > > lot
> > > > > > > > > > > > of
> > > > > > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > > > > > approach
> > > > > > > > > > > will
> > > > > > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > > > > > concurrent
> > > > > > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > > > > > bring
> > > > > > > > > > side
> > > > > > > > > > > > > effects?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > > > > > without
> > > > > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > > > > >
> > > > > > > > > > > > > - Sijie
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the feedback Michael.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > > > > > affect the
> > > > > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is a good question. There isn't a clear answer in the
> > > > > > > > GitHub
> > > > > > > > > > > > Actions
> > > > > > > > > > > > > > Cache documentation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The documentation is
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > > > > > > > .
> > > > > > > > > > > > > > Based on this document and some testing, I have made these
> > > > > > > > > > > conclusions:
> > > > > > > > > > > > > > For GitHub Actions Cache, pull requests get executed in the
> > > > > > > > context
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > forked repository.
> > > > > > > > > > > > > > The workflow triggered by a pull request event can only update
> > > > > > > > it's
> > > > > > > > > > > own
> > > > > > > > > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > > > > > > > > If there's a cache miss, the entry will get written to the
> > > > > > > > cache of
> > > > > > > > > > > the
> > > > > > > > > > > > > > forked repository. If the PR could to write to the upstream
> > > > > > > > cache,
> > > > > > > > > > it
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > be a security issue since this would be vulnerable to cache
> > > > > > > > > > poisoning
> > > > > > > > > > > > > > attacks. Each repository has a 5GB quota for writes. The
> > > > > > > > entries
> > > > > > > > > > are
> > > > > > > > > > > > kept
> > > > > > > > > > > > > > up to 7 days.
> > > > > > > > > > > > > > The performance is fairly good. Loading docker images from the
> > > > > > > > > > > > repository
> > > > > > > > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> > > > > > > > (the
> > > > > > > > > > > > > > performance of the GHA cache is most likely higher since this
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > > > > > throughput for docker load / docker save)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If a single repository has a lot of concurrent jobs, it could
> > > > > > > > start
> > > > > > > > > > > > > > evicting caches.
> > > > > > > > > > > > > > However that isn't likely to happen with the way Pulsar is
> > > > > > > > > > developed
> > > > > > > > > > > > > since
> > > > > > > > > > > > > > pull requests are created from personal forks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > > > > > reference
> > > > > > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > > > > > assuming
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > > > > > completion,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > > > > > I
> > > > > > > > > > > > haven't
> > > > > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> > > > > > > > jobs
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > matrix if one fails. Other parallel flows would run to
> > > > > > > > completion
> > > > > > > > > > by
> > > > > > > > > > > > > > default.
> > > > > > > > > > > > > > In the prototype, I have added a Github script step to each
> > > > > > > > job to
> > > > > > > > > > > > cancel
> > > > > > > > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The prototype follows a "fail fast" design. When a failure
> > > > > > > > occurs,
> > > > > > > > > > > fail
> > > > > > > > > > > > > > fast and don't continue with other jobs.
> > > > > > > > > > > > > > The benefit of this is that it reduces resource consumption.
> > > > > > > > This
> > > > > > > > > > > helps
> > > > > > > > > > > > > > keep the build queue short.
> > > > > > > > > > > > > > When the build queue is short, developers get quick feedback
> > > > > > > > from
> > > > > > > > > > CI.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > > > > > > > > *I'm hoping to start a separate document on low level details
> > > > > > > > when
> > > > > > > > > > > > there
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > a high level acceptance of the proposed "Changes to GitHub
> > > > > > > > Actions
> > > > > > > > > > > > based
> > > > > > > > > > > > > > Pulsar CI".*
> > > > > > > > > > > > > > Together we can make this happen. We need decisions too. This
> > > > > > > > > > > proposal
> > > > > > > > > > > > > > cannot stay as a draft forever.
> > > > > > > > > > > > > > I'm looking forward to hearing from the Pulsar community,
> > > > > > > > Pulsar
> > > > > > > > > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > BR,
> > > > > > > > > > > > > > -Lari
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > > > > > > > > mikemarsh17@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > > > > > > > > overall,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > looks good to me.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > > > > > affect the
> > > > > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > > > > > reference
> > > > > > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > > > > > assuming
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > > > > > completion,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > > > > > I
> > > > > > > > > > > > haven't
> > > > > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> > > > > > > > community
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> > > > > > > > has
> > > > > > > > > > > gone
> > > > > > > > > > > > > > > forward
> > > > > > > > > > > > > > > > > based on your feedback. Here are some updates about the
> > > > > > > > work.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > > > > There's a *detailed status update in the document about a
> > > > > > > > > > > > prototype
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> > > > > > > > lot of
> > > > > > > > > > > > > > > improvements
> > > > > > > > > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > > > > > > > > technical
> > > > > > > > > > > > > > > > > obstacles.
> > > > > > > > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> > > > > > > > docker
> > > > > > > > > > > > > images.
> > > > > > > > > > > > > > A
> > > > > > > > > > > > > > > > lot
> > > > > > > > > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > > > > > > > > build
> > > > > > > > > > > to
> > > > > > > > > > > > > > > enable
> > > > > > > > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> > > > > > > > for all
> > > > > > > > > > > PR
> > > > > > > > > > > > > > > reviews
> > > > > > > > > > > > > > > > > and feedback.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> > > > > > > > Pulsar
> > > > > > > > > > > CI"
> > > > > > > > > > > > > work
> > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > > > > > > > > build
> > > > > > > > > > > > and
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > > > > > > > > productivity*
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> > > > > > > > feedback.
> > > > > > > > > > > The
> > > > > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > > > > > > > > resource
> > > > > > > > > > > > > > > > consumption
> > > > > > > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > > > > > > > > observed
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > work began.
> > > > > > > > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> > > > > > > > build,
> > > > > > > > > > the
> > > > > > > > > > > > > > resource
> > > > > > > > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > > > > > > > *This is about 60% reduction in resource consumption.*
> > > > > > > > The
> > > > > > > > > > > whole
> > > > > > > > > > > > > > > pipeline
> > > > > > > > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Here's a breakdown of the duration (resource
> > > > > > > > consumption) of
> > > > > > > > > > > each
> > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > job
> > > > > > > > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > > > > > > > > 0:06:45
> > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> > > > > > > > 0:16:08
> > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> > > > > > > > 0:37:03
> > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > > > > > > > > 0:14:09
> > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > > > > > > > > 0:29:55
> > > > > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > > > > > > > > 0:38:32
> > > > > > > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > > > > > > > 7:08:57
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > However, this is not a showstopper since there are ways
> > > > > > > > to
> > > > > > > > > > > > address
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > > > > > > > There is a separate PIP for changing the way to handle
> > > > > > > > flaky
> > > > > > > > > > > > tests.
> > > > > > > > > > > > > > You
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> > > > > > > > based
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > CI"
> > > > > > > > > > > > > > > > > document's header.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > > > > . *It also contains more details of the prototype that
> > > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > > > > > > successfully completed.*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2) *Please share your feedback and suggest a way
> > > > > > > > forward.*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > > > > > > > consuming
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > majority
> > > > > > > > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > > > > > > > github.com/apache
> > > > > > > > > > > > > > > > > projects.
> > > > > > > > > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > > > > > > > > demand
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > improve
> > > > > > > > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > >
> > > > > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > asap.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > > > > > > > > time
> > > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > > Runner
> > > > > > > > > > > > > > > > > > is occupied. I observed the workflow runs for handling
> > > > > > > > a
> > > > > > > > > > > single
> > > > > > > > > > > > > > Pull
> > > > > > > > > > > > > > > > > > Request (in my personal fork) and these were the
> > > > > > > > running
> > > > > > > > > > > > > durations:
> > > > > > > > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > > > > > > > > Actions
> > > > > > > > > > > > > > > Runners
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > > > > > > > request to
> > > > > > > > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > > > > > > > is
> > > > > > > > > > very
> > > > > > > > > > > > > high,
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > leads to the build queue to grow and take a long time
> > > > > > > > to
> > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > last
> > > > > > > > > > > > > > 3
> > > > > > > > > > > > > > > > > > months. During this period I worked on a few
> > > > > > > > experiments.
> > > > > > > > > > The
> > > > > > > > > > > > > > > learnings
> > > > > > > > > > > > > > > > > > from the past experiments are documented at a high
> > > > > > > > level in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > > > > > > > CI"
> > > > > > > > > > > > > document
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > Google doc:*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > > > > > > > based
> > > > > > > > > > > on
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > feedback asap. If there's already a similar effort
> > > > > > > > > > ongoing, I
> > > > > > > > > > > > > hope
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > join efforts.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <lh...@apache.org>.
Merging the PR is blocked by https://github.com/apache/pulsar/issues/14951 .
Pulsar SQL doesn't work with Java 11.0.14.1 version. It fails with this error message:
Exception in thread "main" java.lang.IllegalArgumentException: Cannot parse version 11.0.14.1
at io.prestosql.server.JavaVersion.parse(JavaVersion.java:76)
at io.prestosql.server.PrestoSystemRequirements.verifyJavaVersion(PrestoSystemRequirements.java:102)
at io.prestosql.server.PrestoSystemRequirements.verifyJvmRequirements(PrestoSystemRequirements.java:45)
at io.prestosql.server.PrestoServer.run(PrestoServer.java:78)
at io.prestosql.$gen.Presto_332____20220330_100314_1.run(Unknown Source)
at io.prestosql.server.PrestoServer.main(PrestoServer.java:72)
I'll apply a workaround to unblock CI.
-Lari
On 2022/03/30 06:52:38 Lari Hotari wrote:
> Thank you for the reviews and feedback. I have started making the switch to the new refactored Pulsar CI.
>
> Merging new PRs are blocked until the switch is ready. The reason for this is that I have merged https://github.com/apache/pulsar/pull/14939 preparing for merging https://github.com/apache/pulsar/pull/14819 .
>
> The GitHub Actions "required checks" change in the refactored Pulsar CI and there can be only one effective set of "required checks" for a branch.
>
> After the new Pulsar CI workflow PR has been merged, each in-progress PR has to be closed & immediately reopened to pick up the new workflow and the PR build has to run through the new workflow. Another way to pick up the new workflow is to rebase the PR (or merge master branch changes to it).
>
> Please let me know if you experience any issues with the new Pulsar CI workflow. I'll be on the #testing channel on Pulsar Slack too.
>
> -Lari
>
> On 2022/03/29 15:43:31 Michael Marshall wrote:
> > Great work, Lari! It's great news that GitHub's new feature helps this
> > valuable work move forward. I look forward to seeing your PR merged,
> > and I am happy to help resolve any issues that might pop up.
> >
> > Thanks,
> > Michael
> >
> > On Tue, Mar 29, 2022 at 7:55 AM Lari Hotari <lh...@apache.org> wrote:
> > >
> > > The PR has sufficient reviews, and I'll proceed with merging it today or tomorrow.
> > > Please provide feedback now if you want to do that before the PR is merged.
> > >
> > > Thanks!
> > >
> > > -Lari
> > >
> > > On 2022/03/28 20:05:14 Lari Hotari wrote:
> > > > The PR https://github.com/apache/pulsar/pull/14819 is now ready for review.
> > > >
> > > > The changes in the PR now use GitHub Actions Artifacts for sharing binary files (such as docker images) between the build steps. This saves a lot of GitHub Actions VM resources since the docker images are built once and shared in downstream jobs.
> > > > GitHub Actions Artifacts are meant to be used for sharing data between the jobs in a GitHub Actions workflow [1].
> > > >
> > > > I'm looking forward to your review and feedback on https://github.com/apache/pulsar/pull/14819 .
> > > >
> > > > BR,
> > > >
> > > > -Lari
> > > >
> > > > References:
> > > > [1] GitHub Actions: Storing workflow data as artifacts - https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
> > > >
> > > > On 2022/03/23 10:46:10 Lari Hotari wrote:
> > > > > I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
> > > > >
> > > > > BR,
> > > > > -Lari
> > > > >
> > > > > On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> > > > > > Lari,
> > > > > >
> > > > > > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
> > > > > >
> > > > > > > I have resumed work to improve our GitHub Actions based Pulsar CI.
> > > > > > >
> > > > > > > Last year, I worked on a proof-of-concept which significantly reduced the
> > > > > > > resource consumption and improved the usability of the build by combining
> > > > > > > multiple workflows into a single larger workflow.
> > > > > > >
> > > > > > > The showstopper a year ago was the lack of being able to re-run a single
> > > > > > > failed job in a larger workflow.
> > > > > > > GitHub has since then delivered this feature and no showstoppers are
> > > > > > > present.
> > > > > > >
> > > > > > > I have been posting updates to
> > > > > > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> > > > > > > about the progress.
> > > > > > > I have rebased the changes from last year's PoC, and I'm finalizing and
> > > > > > > testing the changes in my fork under
> > > > > > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> > > > > > > apache/pulsar, when the refactoring is ready.
> > > > > > >
> > > > > >
> > > > > > This is great news !
> > > > > >
> > > > > > Looking forward to your patch
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > >
> > > > > >
> > > > > > > -Lari
> > > > > > >
> > > > > > > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > > > > > > The prototype has demonstrated about 60% reduction in
> > > > > > > > resource consumption.
> > > > > > > >
> > > > > > > > It is hard to quantify. Merging them into one large workflow can result
> > > > > > > in
> > > > > > > > more failures. Re-running those failures can consume resources as well.
> > > > > > > >
> > > > > > > > > Isn't it urgent to resolve it?
> > > > > > > >
> > > > > > > > I think we are in a stage that gives us breathing room to fix flaky tests
> > > > > > > > and solve other problems, no?
> > > > > > > > I don't mean we stop the effort here. I mean we have other enhancements
> > > > > > > > that we can do to improve the situation.
> > > > > > > > Once we get into a position where the flakiness is reduced, we can merge
> > > > > > > > them into one workflow.
> > > > > > > >
> > > > > > > > - Sijie
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > > > >
> > > > > > > > > Thanks for the feedback Sijie.
> > > > > > > > >
> > > > > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > > > > objection,
> > > > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > > > If people raise concerns, please address the concerns.
> > > > > > > > >
> > > > > > > > > You and Ali have raised concerns about changing the existing GitHub
> > > > > > > Actions
> > > > > > > > > workflows in a way where multiple workflows would be combined to a
> > > > > > > single
> > > > > > > > > workflow. Before proceeding, there is a need to address the concerns.
> > > > > > > We
> > > > > > > > > might end up with a completely different type of solution of what has
> > > > > > > been
> > > > > > > > > proposed initially. :)
> > > > > > > > >
> > > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > > > > workflows
> > > > > > > > > > into one giant workflow.
> > > > > > > > >
> > > > > > > > > I agree that addressing flaky tests is favorable. The main reason for
> > > > > > > PIP
> > > > > > > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > > > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > > > > > > > 2) Reduce lead times for Pull Request feedback
> > > > > > > > > We cannot ignore these problems. If we don't change anything, the
> > > > > > > problems
> > > > > > > > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > > > > > > > resource consumption. Measuring the lead times hasn't been done in the
> > > > > > > > > prototype, but since the reason for long lead times has been long build
> > > > > > > > > queues due to excessive resource consumption, it's likely that the lead
> > > > > > > > > times would be reduced.
> > > > > > > > >
> > > > > > > > > I know that switching to a single workflow isn't the only solution to
> > > > > > > the
> > > > > > > > > above problems. I had a discussion with Ali. He recommended reducing
> > > > > > > the
> > > > > > > > > modules in Pulsar repository (PIP-62), reducing the docker container
> > > > > > > size
> > > > > > > > > and improving the Pulsar Broker unit test harness so that tests using
> > > > > > > it
> > > > > > > > > would be less flaky and that it would be easier to fix the issues in
> > > > > > > > > failing test when there would be better information about what was the
> > > > > > > > > state problem that caused the test to fail.
> > > > > > > > >
> > > > > > > > > As mentioned in the earlier email about the optimizations in the
> > > > > > > Pulsar CI
> > > > > > > > > refactoring prototype, the main benefits come from reusing binary
> > > > > > > artifacts
> > > > > > > > > from previous build stages so that each job doesn't have to build
> > > > > > > > > everything from scratch. This becomes irrelevant when the build is very
> > > > > > > > > fast and there isn't a benefit of reusing artifacts.
> > > > > > > > > This means that it's possible to resolve the resource consumption
> > > > > > > problem
> > > > > > > > > of Pulsar PR builds in the way that Ali is recommending, without
> > > > > > > switching
> > > > > > > > > from multiple workflows to a single workflow that can reuse binary
> > > > > > > > > artifacts in the build.
> > > > > > > > >
> > > > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > > > > more
> > > > > > > > > > on solving flaky tests.
> > > > > > > > > > Consider merging them into one workflow when the tests are in a
> > > > > > > better
> > > > > > > > > > situation.
> > > > > > > > >
> > > > > > > > > Makes sense for minimizing the risk of change, but we cannot just wait
> > > > > > > for
> > > > > > > > > things to fix themselves.
> > > > > > > > > How long will other Apache projects tolerate the resource consumption
> > > > > > > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> > > > > > > For
> > > > > > > > > example,
> > > > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > .
> > > > > > > > > Isn't it urgent to resolve it?
> > > > > > > > >
> > > > > > > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> > > > > > > CI"
> > > > > > > > > based on the community feedback in the upcoming days. That might mean
> > > > > > > that
> > > > > > > > > the current solution is pivoted. The goal is to solve the problems of
> > > > > > > high
> > > > > > > > > resource consumption and long lead time for PR build in Pulsar CI.
> > > > > > > Please
> > > > > > > > > continue to provide feedback so that we get a revisited plan together
> > > > > > > that
> > > > > > > > > addresses these problems.Thank you!
> > > > > > > > >
> > > > > > > > > BR,
> > > > > > > > > -Lari
> > > > > > > > >
> > > > > > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > > > > GitHub
> > > > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > > >
> > > > > > > > > > I see what you referred to before now. This can be easily done. I
> > > > > > > (or any
> > > > > > > > > > other committer) can do it for you.
> > > > > > > > > >
> > > > > > > > > > There is no real blocker for you to continue work even there are
> > > > > > > concerns
> > > > > > > > > > or it is not listed in the PIP.
> > > > > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > > > > objection,
> > > > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > > > If people raise concerns, please address the concerns.
> > > > > > > > > >
> > > > > > > > > > > The reason why
> > > > > > > > > > re-runs happen currently is because of the high flakiness of tests.
> > > > > > > > > >
> > > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > > > > workflows
> > > > > > > > > > into one giant workflow.
> > > > > > > > > > It is not about "No pain, no gain". The community has suffered a lot
> > > > > > > with
> > > > > > > > > > giant workflow before.
> > > > > > > > > > There were a lot of committers and contributors working hard to
> > > > > > > split one
> > > > > > > > > > giant workflow into multiple
> > > > > > > > > > current workflows. Unless there is really strong evidence that
> > > > > > > merging
> > > > > > > > > them
> > > > > > > > > > back to one will improve
> > > > > > > > > > the entire CI experience, I will still have concerns about one giant
> > > > > > > > > > workflow approach.
> > > > > > > > > >
> > > > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > > > > more
> > > > > > > > > > on solving flaky tests.
> > > > > > > > > > Consider merging them into one workflow when the tests are in a
> > > > > > > better
> > > > > > > > > > situation.
> > > > > > > > > >
> > > > > > > > > > > This solution would also require disabling
> > > > > > > > > > required status checks
> > > > > > > > > >
> > > > > > > > > > I don't think it is a good idea to disable status checks. We can
> > > > > > > consider
> > > > > > > > > > running "dark mode" but it will just overload the action quota.
> > > > > > > > > >
> > > > > > > > > > Another alternative is to mirror the pull requests into another
> > > > > > > Github
> > > > > > > > > > account to test that and get more concrete statistics on the
> > > > > > > flakiness
> > > > > > > > > rate
> > > > > > > > > > of one workflow approach.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for the feedback, Sijie.
> > > > > > > > > > >
> > > > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > > > > there
> > > > > > > > > > > are
> > > > > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > > > > However, I am not sure how much it will help if you run all
> > > > > > > workflows
> > > > > > > > > > in
> > > > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > > >
> > > > > > > > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > > > > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > > > > > > > Addressing flakiness continues to be top-priority. Now that the
> > > > > > > Pulsar
> > > > > > > > > CI
> > > > > > > > > > > workflow prototype is finished, I'll be focusing more in the other
> > > > > > > > > draft
> > > > > > > > > > > PIP, "Changes to flaky test handling".
> > > > > > > > > > > We as a community should address the critical problem that the
> > > > > > > current
> > > > > > > > > > > retry solution has: it can mask bugs in production code and make
> > > > > > > the
> > > > > > > > > > build
> > > > > > > > > > > pass and allow changes to be merged that cause regressions.
> > > > > > > > > > > It's a false sense of security what the green builds after all the
> > > > > > > > > > retries
> > > > > > > > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > > > > > > > addressing this.
> > > > > > > > > > >
> > > > > > > > > > > If something doesn't work, it can be adapted and improved. Changes
> > > > > > > can
> > > > > > > > > be
> > > > > > > > > > > rolled back and revisited when things go worse. We also need a
> > > > > > > leap of
> > > > > > > > > > > faith.
> > > > > > > > > > > "No pain, no gain", like any change, it will be painful at first,
> > > > > > > but
> > > > > > > > > we
> > > > > > > > > > > will get over the bump.
> > > > > > > > > > >
> > > > > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > > > > not a
> > > > > > > > > > lot
> > > > > > > > > > > of
> > > > > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > > > > approach
> > > > > > > > > > will
> > > > > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > > > > concurrent
> > > > > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > > > > bring
> > > > > > > > > side
> > > > > > > > > > > > effects?
> > > > > > > > > > >
> > > > > > > > > > > One possibility to address this is to introduce the new workflow
> > > > > > > in a
> > > > > > > > > > mode
> > > > > > > > > > > where you need to opt-in to the new workflow in some way.
> > > > > > > > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > > > > > > > It might be possible to configure the existing workflow and this
> > > > > > > new
> > > > > > > > > > > workflow in a way where some condition (for example whitelisted
> > > > > > > github
> > > > > > > > > > user
> > > > > > > > > > > name or a certain keyword in the PR title/description) chooses
> > > > > > > either
> > > > > > > > > one
> > > > > > > > > > > for the Pull request. This solution would also require disabling
> > > > > > > > > > > required status checks ("Require status checks to pass before
> > > > > > > merging"
> > > > > > > > > > > feature in GitHub branch protection rules) since the names of the
> > > > > > > > > checks
> > > > > > > > > > > would be different.
> > > > > > > > > > >
> > > > > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > > > > without
> > > > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > > >
> > > > > > > > > > > This is what I have been doing. All individual optimizations have
> > > > > > > > > already
> > > > > > > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > > > > > > > have
> > > > > > > > > > > already been merged.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > > > > > > There's also 2 build related PRs from yesterday which haven't been
> > > > > > > > > merged
> > > > > > > > > > > yet:
> > > > > > > > > > > Fix Maven download issues (ported from the prototype to our
> > > > > > > existing
> > > > > > > > > > Pulsar
> > > > > > > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > > > > > > improve Maven module build order (required for more efficient
> > > > > > > builds
> > > > > > > > > that
> > > > > > > > > > > selectively build required artifacts):
> > > > > > > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > > > > > > >
> > > > > > > > > > > There aren't many optimizations left that could be ported from the
> > > > > > > > > > > prototype to the existing workflow. There are a few, but the
> > > > > > > impact is
> > > > > > > > > > > minor.
> > > > > > > > > > > The reason for this is that the optimization with the greatest
> > > > > > > impact
> > > > > > > > > are
> > > > > > > > > > > the ones that build a binary artifacts (maven libs, docker images)
> > > > > > > once
> > > > > > > > > > and
> > > > > > > > > > > share it with the downstream jobs in the pipeline.
> > > > > > > > > > > Applying this type of solution has certain challenges when there
> > > > > > > are
> > > > > > > > > > > multiple separate workflow. Sharing binary artifacts to other
> > > > > > > workflows
> > > > > > > > > > > would require that the workflow to reuse the artifacts gets
> > > > > > > triggered
> > > > > > > > > by
> > > > > > > > > > > the workflow that produced the artifacts. This wouldn't be secure
> > > > > > > or
> > > > > > > > > > > practical for handling pull requests. Triggering a workflow
> > > > > > > explicitly
> > > > > > > > > > > would require a token from the main repository and using that for
> > > > > > > pull
> > > > > > > > > > > request builds would be a serious security vulnerability. (more
> > > > > > > details
> > > > > > > > > > > about the GitHub Actions security model in
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > > > > > > )
> > > > > > > > > > >
> > > > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > > > > GitHub
> > > > > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> > > > > > > wiki
> > > > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > > > >
> > > > > > > > > > > -Lari
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > This is good progress. However, my main concern is still merging
> > > > > > > all
> > > > > > > > > > > > workflows into one giant workflow.
> > > > > > > > > > > >
> > > > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > > > > there
> > > > > > > > > > > are
> > > > > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > > > > However, I am not sure how much it will help if you run all
> > > > > > > workflows
> > > > > > > > > > in
> > > > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > > > >
> > > > > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > > > > not a
> > > > > > > > > > lot
> > > > > > > > > > > of
> > > > > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > > > > approach
> > > > > > > > > > will
> > > > > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > > > > concurrent
> > > > > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > > > > bring
> > > > > > > > > side
> > > > > > > > > > > > effects?
> > > > > > > > > > > >
> > > > > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > > > > without
> > > > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > > > >
> > > > > > > > > > > > - Sijie
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the feedback Michael.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > > > > affect the
> > > > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is a good question. There isn't a clear answer in the
> > > > > > > GitHub
> > > > > > > > > > > Actions
> > > > > > > > > > > > > Cache documentation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The documentation is
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > > > > > > .
> > > > > > > > > > > > > Based on this document and some testing, I have made these
> > > > > > > > > > conclusions:
> > > > > > > > > > > > > For GitHub Actions Cache, pull requests get executed in the
> > > > > > > context
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > forked repository.
> > > > > > > > > > > > > The workflow triggered by a pull request event can only update
> > > > > > > it's
> > > > > > > > > > own
> > > > > > > > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > > > > > > > If there's a cache miss, the entry will get written to the
> > > > > > > cache of
> > > > > > > > > > the
> > > > > > > > > > > > > forked repository. If the PR could to write to the upstream
> > > > > > > cache,
> > > > > > > > > it
> > > > > > > > > > > > would
> > > > > > > > > > > > > be a security issue since this would be vulnerable to cache
> > > > > > > > > poisoning
> > > > > > > > > > > > > attacks. Each repository has a 5GB quota for writes. The
> > > > > > > entries
> > > > > > > > > are
> > > > > > > > > > > kept
> > > > > > > > > > > > > up to 7 days.
> > > > > > > > > > > > > The performance is fairly good. Loading docker images from the
> > > > > > > > > > > repository
> > > > > > > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> > > > > > > (the
> > > > > > > > > > > > > performance of the GHA cache is most likely higher since this
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > > > > > throughput for docker load / docker save)
> > > > > > > > > > > > >
> > > > > > > > > > > > > If a single repository has a lot of concurrent jobs, it could
> > > > > > > start
> > > > > > > > > > > > > evicting caches.
> > > > > > > > > > > > > However that isn't likely to happen with the way Pulsar is
> > > > > > > > > developed
> > > > > > > > > > > > since
> > > > > > > > > > > > > pull requests are created from personal forks.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > > > > reference
> > > > > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > > > > assuming
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > > > > completion,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > > > > I
> > > > > > > > > > > haven't
> > > > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> > > > > > > jobs
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > matrix if one fails. Other parallel flows would run to
> > > > > > > completion
> > > > > > > > > by
> > > > > > > > > > > > > default.
> > > > > > > > > > > > > In the prototype, I have added a Github script step to each
> > > > > > > job to
> > > > > > > > > > > cancel
> > > > > > > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > > > > > > Here's an example:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > > > > > > >
> > > > > > > > > > > > > The prototype follows a "fail fast" design. When a failure
> > > > > > > occurs,
> > > > > > > > > > fail
> > > > > > > > > > > > > fast and don't continue with other jobs.
> > > > > > > > > > > > > The benefit of this is that it reduces resource consumption.
> > > > > > > This
> > > > > > > > > > helps
> > > > > > > > > > > > > keep the build queue short.
> > > > > > > > > > > > > When the build queue is short, developers get quick feedback
> > > > > > > from
> > > > > > > > > CI.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > > > > > > > *I'm hoping to start a separate document on low level details
> > > > > > > when
> > > > > > > > > > > there
> > > > > > > > > > > > is
> > > > > > > > > > > > > a high level acceptance of the proposed "Changes to GitHub
> > > > > > > Actions
> > > > > > > > > > > based
> > > > > > > > > > > > > Pulsar CI".*
> > > > > > > > > > > > > Together we can make this happen. We need decisions too. This
> > > > > > > > > > proposal
> > > > > > > > > > > > > cannot stay as a draft forever.
> > > > > > > > > > > > > I'm looking forward to hearing from the Pulsar community,
> > > > > > > Pulsar
> > > > > > > > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > > > > > > > >
> > > > > > > > > > > > > BR,
> > > > > > > > > > > > > -Lari
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > > > > > > > mikemarsh17@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > > > > > > > overall,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > looks good to me.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > > > > affect the
> > > > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > > > > reference
> > > > > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > > > > assuming
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > > > > completion,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > > > > I
> > > > > > > > > > > haven't
> > > > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> > > > > > > community
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> > > > > > > has
> > > > > > > > > > gone
> > > > > > > > > > > > > > forward
> > > > > > > > > > > > > > > > based on your feedback. Here are some updates about the
> > > > > > > work.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > > > There's a *detailed status update in the document about a
> > > > > > > > > > > prototype
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> > > > > > > lot of
> > > > > > > > > > > > > > improvements
> > > > > > > > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > > > > > > > technical
> > > > > > > > > > > > > > > > obstacles.
> > > > > > > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> > > > > > > docker
> > > > > > > > > > > > images.
> > > > > > > > > > > > > A
> > > > > > > > > > > > > > > lot
> > > > > > > > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > > > > > > > build
> > > > > > > > > > to
> > > > > > > > > > > > > > enable
> > > > > > > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> > > > > > > for all
> > > > > > > > > > PR
> > > > > > > > > > > > > > reviews
> > > > > > > > > > > > > > > > and feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> > > > > > > Pulsar
> > > > > > > > > > CI"
> > > > > > > > > > > > work
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > > > > > > > build
> > > > > > > > > > > and
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > > > > > > > productivity*
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> > > > > > > feedback.
> > > > > > > > > > The
> > > > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > > > > > > > resource
> > > > > > > > > > > > > > > consumption
> > > > > > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > > > > > > > observed
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > work began.
> > > > > > > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> > > > > > > build,
> > > > > > > > > the
> > > > > > > > > > > > > resource
> > > > > > > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > > > > > > *This is about 60% reduction in resource consumption.*
> > > > > > > The
> > > > > > > > > > whole
> > > > > > > > > > > > > > pipeline
> > > > > > > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Here's a breakdown of the duration (resource
> > > > > > > consumption) of
> > > > > > > > > > each
> > > > > > > > > > > > > build
> > > > > > > > > > > > > > > job
> > > > > > > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > > > > > > > 0:06:45
> > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> > > > > > > 0:16:08
> > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> > > > > > > 0:37:03
> > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > > > > > > > 0:14:09
> > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > > > > > > > 0:29:55
> > > > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > > > > > > > 0:38:32
> > > > > > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > > > > > > 7:08:57
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > However, this is not a showstopper since there are ways
> > > > > > > to
> > > > > > > > > > > address
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > > > > > > There is a separate PIP for changing the way to handle
> > > > > > > flaky
> > > > > > > > > > > tests.
> > > > > > > > > > > > > You
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> > > > > > > based
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > CI"
> > > > > > > > > > > > > > > > document's header.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > > > . *It also contains more details of the prototype that
> > > > > > > has
> > > > > > > > > been
> > > > > > > > > > > > > > > > successfully completed.*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2) *Please share your feedback and suggest a way
> > > > > > > forward.*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > > > > > > consuming
> > > > > > > > > > > the
> > > > > > > > > > > > > > > majority
> > > > > > > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > > > > > > github.com/apache
> > > > > > > > > > > > > > > > projects.
> > > > > > > > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > > > > > > > demand
> > > > > > > > > > > to
> > > > > > > > > > > > > > > improve
> > > > > > > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > > > > > > <
> > > > > > > > > > > >
> > > > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > asap.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > > > > > > > time
> > > > > > > > > > > > that a
> > > > > > > > > > > > > > > > Runner
> > > > > > > > > > > > > > > > > is occupied. I observed the workflow runs for handling
> > > > > > > a
> > > > > > > > > > single
> > > > > > > > > > > > > Pull
> > > > > > > > > > > > > > > > > Request (in my personal fork) and these were the
> > > > > > > running
> > > > > > > > > > > > durations:
> > > > > > > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > > > > > > > Actions
> > > > > > > > > > > > > > Runners
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > > > > > > request to
> > > > > > > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > > > > > > is
> > > > > > > > > very
> > > > > > > > > > > > high,
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > leads to the build queue to grow and take a long time
> > > > > > > to
> > > > > > > > > > > process.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > last
> > > > > > > > > > > > > 3
> > > > > > > > > > > > > > > > > months. During this period I worked on a few
> > > > > > > experiments.
> > > > > > > > > The
> > > > > > > > > > > > > > learnings
> > > > > > > > > > > > > > > > > from the past experiments are documented at a high
> > > > > > > level in
> > > > > > > > > > the
> > > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > > > > > > CI"
> > > > > > > > > > > > document
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > Google doc:*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > > > > > > based
> > > > > > > > > > on
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > feedback asap. If there's already a similar effort
> > > > > > > > > ongoing, I
> > > > > > > > > > > > hope
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > join efforts.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <lh...@apache.org>.
Thank you for the reviews and feedback. I have started making the switch to the new refactored Pulsar CI.
Merging new PRs are blocked until the switch is ready. The reason for this is that I have merged https://github.com/apache/pulsar/pull/14939 preparing for merging https://github.com/apache/pulsar/pull/14819 .
The GitHub Actions "required checks" change in the refactored Pulsar CI and there can be only one effective set of "required checks" for a branch.
After the new Pulsar CI workflow PR has been merged, each in-progress PR has to be closed & immediately reopened to pick up the new workflow and the PR build has to run through the new workflow. Another way to pick up the new workflow is to rebase the PR (or merge master branch changes to it).
Please let me know if you experience any issues with the new Pulsar CI workflow. I'll be on the #testing channel on Pulsar Slack too.
-Lari
On 2022/03/29 15:43:31 Michael Marshall wrote:
> Great work, Lari! It's great news that GitHub's new feature helps this
> valuable work move forward. I look forward to seeing your PR merged,
> and I am happy to help resolve any issues that might pop up.
>
> Thanks,
> Michael
>
> On Tue, Mar 29, 2022 at 7:55 AM Lari Hotari <lh...@apache.org> wrote:
> >
> > The PR has sufficient reviews, and I'll proceed with merging it today or tomorrow.
> > Please provide feedback now if you want to do that before the PR is merged.
> >
> > Thanks!
> >
> > -Lari
> >
> > On 2022/03/28 20:05:14 Lari Hotari wrote:
> > > The PR https://github.com/apache/pulsar/pull/14819 is now ready for review.
> > >
> > > The changes in the PR now use GitHub Actions Artifacts for sharing binary files (such as docker images) between the build steps. This saves a lot of GitHub Actions VM resources since the docker images are built once and shared in downstream jobs.
> > > GitHub Actions Artifacts are meant to be used for sharing data between the jobs in a GitHub Actions workflow [1].
> > >
> > > I'm looking forward to your review and feedback on https://github.com/apache/pulsar/pull/14819 .
> > >
> > > BR,
> > >
> > > -Lari
> > >
> > > References:
> > > [1] GitHub Actions: Storing workflow data as artifacts - https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
> > >
> > > On 2022/03/23 10:46:10 Lari Hotari wrote:
> > > > I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
> > > >
> > > > BR,
> > > > -Lari
> > > >
> > > > On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> > > > > Lari,
> > > > >
> > > > > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
> > > > >
> > > > > > I have resumed work to improve our GitHub Actions based Pulsar CI.
> > > > > >
> > > > > > Last year, I worked on a proof-of-concept which significantly reduced the
> > > > > > resource consumption and improved the usability of the build by combining
> > > > > > multiple workflows into a single larger workflow.
> > > > > >
> > > > > > The showstopper a year ago was the lack of being able to re-run a single
> > > > > > failed job in a larger workflow.
> > > > > > GitHub has since then delivered this feature and no showstoppers are
> > > > > > present.
> > > > > >
> > > > > > I have been posting updates to
> > > > > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> > > > > > about the progress.
> > > > > > I have rebased the changes from last year's PoC, and I'm finalizing and
> > > > > > testing the changes in my fork under
> > > > > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> > > > > > apache/pulsar, when the refactoring is ready.
> > > > > >
> > > > >
> > > > > This is great news !
> > > > >
> > > > > Looking forward to your patch
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > >
> > > > > > -Lari
> > > > > >
> > > > > > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > > > > > The prototype has demonstrated about 60% reduction in
> > > > > > > resource consumption.
> > > > > > >
> > > > > > > It is hard to quantify. Merging them into one large workflow can result
> > > > > > in
> > > > > > > more failures. Re-running those failures can consume resources as well.
> > > > > > >
> > > > > > > > Isn't it urgent to resolve it?
> > > > > > >
> > > > > > > I think we are in a stage that gives us breathing room to fix flaky tests
> > > > > > > and solve other problems, no?
> > > > > > > I don't mean we stop the effort here. I mean we have other enhancements
> > > > > > > that we can do to improve the situation.
> > > > > > > Once we get into a position where the flakiness is reduced, we can merge
> > > > > > > them into one workflow.
> > > > > > >
> > > > > > > - Sijie
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > > >
> > > > > > > > Thanks for the feedback Sijie.
> > > > > > > >
> > > > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > > > objection,
> > > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > > If people raise concerns, please address the concerns.
> > > > > > > >
> > > > > > > > You and Ali have raised concerns about changing the existing GitHub
> > > > > > Actions
> > > > > > > > workflows in a way where multiple workflows would be combined to a
> > > > > > single
> > > > > > > > workflow. Before proceeding, there is a need to address the concerns.
> > > > > > We
> > > > > > > > might end up with a completely different type of solution of what has
> > > > > > been
> > > > > > > > proposed initially. :)
> > > > > > > >
> > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > > > workflows
> > > > > > > > > into one giant workflow.
> > > > > > > >
> > > > > > > > I agree that addressing flaky tests is favorable. The main reason for
> > > > > > PIP
> > > > > > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > > > > > > 2) Reduce lead times for Pull Request feedback
> > > > > > > > We cannot ignore these problems. If we don't change anything, the
> > > > > > problems
> > > > > > > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > > > > > > resource consumption. Measuring the lead times hasn't been done in the
> > > > > > > > prototype, but since the reason for long lead times has been long build
> > > > > > > > queues due to excessive resource consumption, it's likely that the lead
> > > > > > > > times would be reduced.
> > > > > > > >
> > > > > > > > I know that switching to a single workflow isn't the only solution to
> > > > > > the
> > > > > > > > above problems. I had a discussion with Ali. He recommended reducing
> > > > > > the
> > > > > > > > modules in Pulsar repository (PIP-62), reducing the docker container
> > > > > > size
> > > > > > > > and improving the Pulsar Broker unit test harness so that tests using
> > > > > > it
> > > > > > > > would be less flaky and that it would be easier to fix the issues in
> > > > > > > > failing test when there would be better information about what was the
> > > > > > > > state problem that caused the test to fail.
> > > > > > > >
> > > > > > > > As mentioned in the earlier email about the optimizations in the
> > > > > > Pulsar CI
> > > > > > > > refactoring prototype, the main benefits come from reusing binary
> > > > > > artifacts
> > > > > > > > from previous build stages so that each job doesn't have to build
> > > > > > > > everything from scratch. This becomes irrelevant when the build is very
> > > > > > > > fast and there isn't a benefit of reusing artifacts.
> > > > > > > > This means that it's possible to resolve the resource consumption
> > > > > > problem
> > > > > > > > of Pulsar PR builds in the way that Ali is recommending, without
> > > > > > switching
> > > > > > > > from multiple workflows to a single workflow that can reuse binary
> > > > > > > > artifacts in the build.
> > > > > > > >
> > > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > > > more
> > > > > > > > > on solving flaky tests.
> > > > > > > > > Consider merging them into one workflow when the tests are in a
> > > > > > better
> > > > > > > > > situation.
> > > > > > > >
> > > > > > > > Makes sense for minimizing the risk of change, but we cannot just wait
> > > > > > for
> > > > > > > > things to fix themselves.
> > > > > > > > How long will other Apache projects tolerate the resource consumption
> > > > > > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> > > > > > For
> > > > > > > > example,
> > > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > .
> > > > > > > > Isn't it urgent to resolve it?
> > > > > > > >
> > > > > > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> > > > > > CI"
> > > > > > > > based on the community feedback in the upcoming days. That might mean
> > > > > > that
> > > > > > > > the current solution is pivoted. The goal is to solve the problems of
> > > > > > high
> > > > > > > > resource consumption and long lead time for PR build in Pulsar CI.
> > > > > > Please
> > > > > > > > continue to provide feedback so that we get a revisited plan together
> > > > > > that
> > > > > > > > addresses these problems.Thank you!
> > > > > > > >
> > > > > > > > BR,
> > > > > > > > -Lari
> > > > > > > >
> > > > > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > > > GitHub
> > > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > >
> > > > > > > > > I see what you referred to before now. This can be easily done. I
> > > > > > (or any
> > > > > > > > > other committer) can do it for you.
> > > > > > > > >
> > > > > > > > > There is no real blocker for you to continue work even there are
> > > > > > concerns
> > > > > > > > > or it is not listed in the PIP.
> > > > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > > > objection,
> > > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > > If people raise concerns, please address the concerns.
> > > > > > > > >
> > > > > > > > > > The reason why
> > > > > > > > > re-runs happen currently is because of the high flakiness of tests.
> > > > > > > > >
> > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > > > workflows
> > > > > > > > > into one giant workflow.
> > > > > > > > > It is not about "No pain, no gain". The community has suffered a lot
> > > > > > with
> > > > > > > > > giant workflow before.
> > > > > > > > > There were a lot of committers and contributors working hard to
> > > > > > split one
> > > > > > > > > giant workflow into multiple
> > > > > > > > > current workflows. Unless there is really strong evidence that
> > > > > > merging
> > > > > > > > them
> > > > > > > > > back to one will improve
> > > > > > > > > the entire CI experience, I will still have concerns about one giant
> > > > > > > > > workflow approach.
> > > > > > > > >
> > > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > > > more
> > > > > > > > > on solving flaky tests.
> > > > > > > > > Consider merging them into one workflow when the tests are in a
> > > > > > better
> > > > > > > > > situation.
> > > > > > > > >
> > > > > > > > > > This solution would also require disabling
> > > > > > > > > required status checks
> > > > > > > > >
> > > > > > > > > I don't think it is a good idea to disable status checks. We can
> > > > > > consider
> > > > > > > > > running "dark mode" but it will just overload the action quota.
> > > > > > > > >
> > > > > > > > > Another alternative is to mirror the pull requests into another
> > > > > > Github
> > > > > > > > > account to test that and get more concrete statistics on the
> > > > > > flakiness
> > > > > > > > rate
> > > > > > > > > of one workflow approach.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for the feedback, Sijie.
> > > > > > > > > >
> > > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > > > there
> > > > > > > > > > are
> > > > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > > > However, I am not sure how much it will help if you run all
> > > > > > workflows
> > > > > > > > > in
> > > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > >
> > > > > > > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > > > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > > > > > > Addressing flakiness continues to be top-priority. Now that the
> > > > > > Pulsar
> > > > > > > > CI
> > > > > > > > > > workflow prototype is finished, I'll be focusing more in the other
> > > > > > > > draft
> > > > > > > > > > PIP, "Changes to flaky test handling".
> > > > > > > > > > We as a community should address the critical problem that the
> > > > > > current
> > > > > > > > > > retry solution has: it can mask bugs in production code and make
> > > > > > the
> > > > > > > > > build
> > > > > > > > > > pass and allow changes to be merged that cause regressions.
> > > > > > > > > > It's a false sense of security what the green builds after all the
> > > > > > > > > retries
> > > > > > > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > > > > > > addressing this.
> > > > > > > > > >
> > > > > > > > > > If something doesn't work, it can be adapted and improved. Changes
> > > > > > can
> > > > > > > > be
> > > > > > > > > > rolled back and revisited when things go worse. We also need a
> > > > > > leap of
> > > > > > > > > > faith.
> > > > > > > > > > "No pain, no gain", like any change, it will be painful at first,
> > > > > > but
> > > > > > > > we
> > > > > > > > > > will get over the bump.
> > > > > > > > > >
> > > > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > > > not a
> > > > > > > > > lot
> > > > > > > > > > of
> > > > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > > > approach
> > > > > > > > > will
> > > > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > > > concurrent
> > > > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > > > bring
> > > > > > > > side
> > > > > > > > > > > effects?
> > > > > > > > > >
> > > > > > > > > > One possibility to address this is to introduce the new workflow
> > > > > > in a
> > > > > > > > > mode
> > > > > > > > > > where you need to opt-in to the new workflow in some way.
> > > > > > > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > > > > > > It might be possible to configure the existing workflow and this
> > > > > > new
> > > > > > > > > > workflow in a way where some condition (for example whitelisted
> > > > > > github
> > > > > > > > > user
> > > > > > > > > > name or a certain keyword in the PR title/description) chooses
> > > > > > either
> > > > > > > > one
> > > > > > > > > > for the Pull request. This solution would also require disabling
> > > > > > > > > > required status checks ("Require status checks to pass before
> > > > > > merging"
> > > > > > > > > > feature in GitHub branch protection rules) since the names of the
> > > > > > > > checks
> > > > > > > > > > would be different.
> > > > > > > > > >
> > > > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > > > without
> > > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > >
> > > > > > > > > > This is what I have been doing. All individual optimizations have
> > > > > > > > already
> > > > > > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > > > > > > have
> > > > > > > > > > already been merged.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > > > > > There's also 2 build related PRs from yesterday which haven't been
> > > > > > > > merged
> > > > > > > > > > yet:
> > > > > > > > > > Fix Maven download issues (ported from the prototype to our
> > > > > > existing
> > > > > > > > > Pulsar
> > > > > > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > > > > > improve Maven module build order (required for more efficient
> > > > > > builds
> > > > > > > > that
> > > > > > > > > > selectively build required artifacts):
> > > > > > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > > > > > >
> > > > > > > > > > There aren't many optimizations left that could be ported from the
> > > > > > > > > > prototype to the existing workflow. There are a few, but the
> > > > > > impact is
> > > > > > > > > > minor.
> > > > > > > > > > The reason for this is that the optimization with the greatest
> > > > > > impact
> > > > > > > > are
> > > > > > > > > > the ones that build a binary artifacts (maven libs, docker images)
> > > > > > once
> > > > > > > > > and
> > > > > > > > > > share it with the downstream jobs in the pipeline.
> > > > > > > > > > Applying this type of solution has certain challenges when there
> > > > > > are
> > > > > > > > > > multiple separate workflow. Sharing binary artifacts to other
> > > > > > workflows
> > > > > > > > > > would require that the workflow to reuse the artifacts gets
> > > > > > triggered
> > > > > > > > by
> > > > > > > > > > the workflow that produced the artifacts. This wouldn't be secure
> > > > > > or
> > > > > > > > > > practical for handling pull requests. Triggering a workflow
> > > > > > explicitly
> > > > > > > > > > would require a token from the main repository and using that for
> > > > > > pull
> > > > > > > > > > request builds would be a serious security vulnerability. (more
> > > > > > details
> > > > > > > > > > about the GitHub Actions security model in
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > > > GitHub
> > > > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> > > > > > wiki
> > > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > > >
> > > > > > > > > > -Lari
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > This is good progress. However, my main concern is still merging
> > > > > > all
> > > > > > > > > > > workflows into one giant workflow.
> > > > > > > > > > >
> > > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > > > there
> > > > > > > > > > are
> > > > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > > > However, I am not sure how much it will help if you run all
> > > > > > workflows
> > > > > > > > > in
> > > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > > >
> > > > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > > > not a
> > > > > > > > > lot
> > > > > > > > > > of
> > > > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > > > approach
> > > > > > > > > will
> > > > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > > > concurrent
> > > > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > > > bring
> > > > > > > > side
> > > > > > > > > > > effects?
> > > > > > > > > > >
> > > > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > > > without
> > > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > > >
> > > > > > > > > > > - Sijie
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for the feedback Michael.
> > > > > > > > > > > >
> > > > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > > > affect the
> > > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > >
> > > > > > > > > > > > This is a good question. There isn't a clear answer in the
> > > > > > GitHub
> > > > > > > > > > Actions
> > > > > > > > > > > > Cache documentation.
> > > > > > > > > > > >
> > > > > > > > > > > > The documentation is
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > > > > > .
> > > > > > > > > > > > Based on this document and some testing, I have made these
> > > > > > > > > conclusions:
> > > > > > > > > > > > For GitHub Actions Cache, pull requests get executed in the
> > > > > > context
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > forked repository.
> > > > > > > > > > > > The workflow triggered by a pull request event can only update
> > > > > > it's
> > > > > > > > > own
> > > > > > > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > > > > > > If there's a cache miss, the entry will get written to the
> > > > > > cache of
> > > > > > > > > the
> > > > > > > > > > > > forked repository. If the PR could to write to the upstream
> > > > > > cache,
> > > > > > > > it
> > > > > > > > > > > would
> > > > > > > > > > > > be a security issue since this would be vulnerable to cache
> > > > > > > > poisoning
> > > > > > > > > > > > attacks. Each repository has a 5GB quota for writes. The
> > > > > > entries
> > > > > > > > are
> > > > > > > > > > kept
> > > > > > > > > > > > up to 7 days.
> > > > > > > > > > > > The performance is fairly good. Loading docker images from the
> > > > > > > > > > repository
> > > > > > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> > > > > > (the
> > > > > > > > > > > > performance of the GHA cache is most likely higher since this
> > > > > > is
> > > > > > > > the
> > > > > > > > > > > > throughput for docker load / docker save)
> > > > > > > > > > > >
> > > > > > > > > > > > If a single repository has a lot of concurrent jobs, it could
> > > > > > start
> > > > > > > > > > > > evicting caches.
> > > > > > > > > > > > However that isn't likely to happen with the way Pulsar is
> > > > > > > > developed
> > > > > > > > > > > since
> > > > > > > > > > > > pull requests are created from personal forks.
> > > > > > > > > > > >
> > > > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > > > reference
> > > > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > > > assuming
> > > > > > > > > > > > that
> > > > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > > > completion,
> > > > > > > > > > > and
> > > > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > > > I
> > > > > > > > > > haven't
> > > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > >
> > > > > > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> > > > > > jobs
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > matrix if one fails. Other parallel flows would run to
> > > > > > completion
> > > > > > > > by
> > > > > > > > > > > > default.
> > > > > > > > > > > > In the prototype, I have added a Github script step to each
> > > > > > job to
> > > > > > > > > > cancel
> > > > > > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > > > > > Here's an example:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > > > > > >
> > > > > > > > > > > > The prototype follows a "fail fast" design. When a failure
> > > > > > occurs,
> > > > > > > > > fail
> > > > > > > > > > > > fast and don't continue with other jobs.
> > > > > > > > > > > > The benefit of this is that it reduces resource consumption.
> > > > > > This
> > > > > > > > > helps
> > > > > > > > > > > > keep the build queue short.
> > > > > > > > > > > > When the build queue is short, developers get quick feedback
> > > > > > from
> > > > > > > > CI.
> > > > > > > > > > > >
> > > > > > > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > > > > > > *I'm hoping to start a separate document on low level details
> > > > > > when
> > > > > > > > > > there
> > > > > > > > > > > is
> > > > > > > > > > > > a high level acceptance of the proposed "Changes to GitHub
> > > > > > Actions
> > > > > > > > > > based
> > > > > > > > > > > > Pulsar CI".*
> > > > > > > > > > > > Together we can make this happen. We need decisions too. This
> > > > > > > > > proposal
> > > > > > > > > > > > cannot stay as a draft forever.
> > > > > > > > > > > > I'm looking forward to hearing from the Pulsar community,
> > > > > > Pulsar
> > > > > > > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > > > > > > >
> > > > > > > > > > > > BR,
> > > > > > > > > > > > -Lari
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > > > > > > mikemarsh17@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > > > > > > overall,
> > > > > > > > > > > it
> > > > > > > > > > > > > looks good to me.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > > > affect the
> > > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > > > reference
> > > > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > > > assuming
> > > > > > > > > > > > that
> > > > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > > > completion,
> > > > > > > > > > > and
> > > > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > > > I
> > > > > > > > > > haven't
> > > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> > > > > > community
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> > > > > > has
> > > > > > > > > gone
> > > > > > > > > > > > > forward
> > > > > > > > > > > > > > > based on your feedback. Here are some updates about the
> > > > > > work.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > > There's a *detailed status update in the document about a
> > > > > > > > > > prototype
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> > > > > > lot of
> > > > > > > > > > > > > improvements
> > > > > > > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > > > > > > technical
> > > > > > > > > > > > > > > obstacles.
> > > > > > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> > > > > > docker
> > > > > > > > > > > images.
> > > > > > > > > > > > A
> > > > > > > > > > > > > > lot
> > > > > > > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > > > > > > build
> > > > > > > > > to
> > > > > > > > > > > > > enable
> > > > > > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> > > > > > for all
> > > > > > > > > PR
> > > > > > > > > > > > > reviews
> > > > > > > > > > > > > > > and feedback.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> > > > > > Pulsar
> > > > > > > > > CI"
> > > > > > > > > > > work
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > > > > > > build
> > > > > > > > > > and
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > > > > > > productivity*
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> > > > > > feedback.
> > > > > > > > > The
> > > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > > > > > > resource
> > > > > > > > > > > > > > consumption
> > > > > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > > > > > > observed
> > > > > > > > > > > > when
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > work began.
> > > > > > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> > > > > > build,
> > > > > > > > the
> > > > > > > > > > > > resource
> > > > > > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > > > > > *This is about 60% reduction in resource consumption.*
> > > > > > The
> > > > > > > > > whole
> > > > > > > > > > > > > pipeline
> > > > > > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here's a breakdown of the duration (resource
> > > > > > consumption) of
> > > > > > > > > each
> > > > > > > > > > > > build
> > > > > > > > > > > > > > job
> > > > > > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > > > > > > 0:06:45
> > > > > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> > > > > > 0:16:08
> > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> > > > > > 0:37:03
> > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > > > > > > 0:14:09
> > > > > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > > > > > > 0:29:55
> > > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > > > > > > 0:38:32
> > > > > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > > > > > 7:08:57
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > However, this is not a showstopper since there are ways
> > > > > > to
> > > > > > > > > > address
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > > > > > There is a separate PIP for changing the way to handle
> > > > > > flaky
> > > > > > > > > > tests.
> > > > > > > > > > > > You
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> > > > > > based
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > CI"
> > > > > > > > > > > > > > > document's header.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > > . *It also contains more details of the prototype that
> > > > > > has
> > > > > > > > been
> > > > > > > > > > > > > > > successfully completed.*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2) *Please share your feedback and suggest a way
> > > > > > forward.*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > > > > > consuming
> > > > > > > > > > the
> > > > > > > > > > > > > > majority
> > > > > > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > > > > > github.com/apache
> > > > > > > > > > > > > > > projects.
> > > > > > > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > > > > > > demand
> > > > > > > > > > to
> > > > > > > > > > > > > > improve
> > > > > > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > asap.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > > > > > > time
> > > > > > > > > > > that a
> > > > > > > > > > > > > > > Runner
> > > > > > > > > > > > > > > > is occupied. I observed the workflow runs for handling
> > > > > > a
> > > > > > > > > single
> > > > > > > > > > > > Pull
> > > > > > > > > > > > > > > > Request (in my personal fork) and these were the
> > > > > > running
> > > > > > > > > > > durations:
> > > > > > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > > > > > > Actions
> > > > > > > > > > > > > Runners
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > > > > > request to
> > > > > > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > > > > > is
> > > > > > > > very
> > > > > > > > > > > high,
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > leads to the build queue to grow and take a long time
> > > > > > to
> > > > > > > > > > process.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > last
> > > > > > > > > > > > 3
> > > > > > > > > > > > > > > > months. During this period I worked on a few
> > > > > > experiments.
> > > > > > > > The
> > > > > > > > > > > > > learnings
> > > > > > > > > > > > > > > > from the past experiments are documented at a high
> > > > > > level in
> > > > > > > > > the
> > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > > > > > CI"
> > > > > > > > > > > document
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > Google doc:*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > > > > > based
> > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > feedback asap. If there's already a similar effort
> > > > > > > > ongoing, I
> > > > > > > > > > > hope
> > > > > > > > > > > > we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > join efforts.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Michael Marshall <mm...@apache.org>.
Great work, Lari! It's great news that GitHub's new feature helps this
valuable work move forward. I look forward to seeing your PR merged,
and I am happy to help resolve any issues that might pop up.
Thanks,
Michael
On Tue, Mar 29, 2022 at 7:55 AM Lari Hotari <lh...@apache.org> wrote:
>
> The PR has sufficient reviews, and I'll proceed with merging it today or tomorrow.
> Please provide feedback now if you want to do that before the PR is merged.
>
> Thanks!
>
> -Lari
>
> On 2022/03/28 20:05:14 Lari Hotari wrote:
> > The PR https://github.com/apache/pulsar/pull/14819 is now ready for review.
> >
> > The changes in the PR now use GitHub Actions Artifacts for sharing binary files (such as docker images) between the build steps. This saves a lot of GitHub Actions VM resources since the docker images are built once and shared in downstream jobs.
> > GitHub Actions Artifacts are meant to be used for sharing data between the jobs in a GitHub Actions workflow [1].
> >
> > I'm looking forward to your review and feedback on https://github.com/apache/pulsar/pull/14819 .
> >
> > BR,
> >
> > -Lari
> >
> > References:
> > [1] GitHub Actions: Storing workflow data as artifacts - https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
> >
> > On 2022/03/23 10:46:10 Lari Hotari wrote:
> > > I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
> > >
> > > BR,
> > > -Lari
> > >
> > > On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> > > > Lari,
> > > >
> > > > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
> > > >
> > > > > I have resumed work to improve our GitHub Actions based Pulsar CI.
> > > > >
> > > > > Last year, I worked on a proof-of-concept which significantly reduced the
> > > > > resource consumption and improved the usability of the build by combining
> > > > > multiple workflows into a single larger workflow.
> > > > >
> > > > > The showstopper a year ago was the lack of being able to re-run a single
> > > > > failed job in a larger workflow.
> > > > > GitHub has since then delivered this feature and no showstoppers are
> > > > > present.
> > > > >
> > > > > I have been posting updates to
> > > > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> > > > > about the progress.
> > > > > I have rebased the changes from last year's PoC, and I'm finalizing and
> > > > > testing the changes in my fork under
> > > > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> > > > > apache/pulsar, when the refactoring is ready.
> > > > >
> > > >
> > > > This is great news !
> > > >
> > > > Looking forward to your patch
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > > > -Lari
> > > > >
> > > > > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > > > > The prototype has demonstrated about 60% reduction in
> > > > > > resource consumption.
> > > > > >
> > > > > > It is hard to quantify. Merging them into one large workflow can result
> > > > > in
> > > > > > more failures. Re-running those failures can consume resources as well.
> > > > > >
> > > > > > > Isn't it urgent to resolve it?
> > > > > >
> > > > > > I think we are in a stage that gives us breathing room to fix flaky tests
> > > > > > and solve other problems, no?
> > > > > > I don't mean we stop the effort here. I mean we have other enhancements
> > > > > > that we can do to improve the situation.
> > > > > > Once we get into a position where the flakiness is reduced, we can merge
> > > > > > them into one workflow.
> > > > > >
> > > > > > - Sijie
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > >
> > > > > > > Thanks for the feedback Sijie.
> > > > > > >
> > > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > > objection,
> > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > If people raise concerns, please address the concerns.
> > > > > > >
> > > > > > > You and Ali have raised concerns about changing the existing GitHub
> > > > > Actions
> > > > > > > workflows in a way where multiple workflows would be combined to a
> > > > > single
> > > > > > > workflow. Before proceeding, there is a need to address the concerns.
> > > > > We
> > > > > > > might end up with a completely different type of solution of what has
> > > > > been
> > > > > > > proposed initially. :)
> > > > > > >
> > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > > workflows
> > > > > > > > into one giant workflow.
> > > > > > >
> > > > > > > I agree that addressing flaky tests is favorable. The main reason for
> > > > > PIP
> > > > > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > > > > > 2) Reduce lead times for Pull Request feedback
> > > > > > > We cannot ignore these problems. If we don't change anything, the
> > > > > problems
> > > > > > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > > > > > resource consumption. Measuring the lead times hasn't been done in the
> > > > > > > prototype, but since the reason for long lead times has been long build
> > > > > > > queues due to excessive resource consumption, it's likely that the lead
> > > > > > > times would be reduced.
> > > > > > >
> > > > > > > I know that switching to a single workflow isn't the only solution to
> > > > > the
> > > > > > > above problems. I had a discussion with Ali. He recommended reducing
> > > > > the
> > > > > > > modules in Pulsar repository (PIP-62), reducing the docker container
> > > > > size
> > > > > > > and improving the Pulsar Broker unit test harness so that tests using
> > > > > it
> > > > > > > would be less flaky and that it would be easier to fix the issues in
> > > > > > > failing test when there would be better information about what was the
> > > > > > > state problem that caused the test to fail.
> > > > > > >
> > > > > > > As mentioned in the earlier email about the optimizations in the
> > > > > Pulsar CI
> > > > > > > refactoring prototype, the main benefits come from reusing binary
> > > > > artifacts
> > > > > > > from previous build stages so that each job doesn't have to build
> > > > > > > everything from scratch. This becomes irrelevant when the build is very
> > > > > > > fast and there isn't a benefit of reusing artifacts.
> > > > > > > This means that it's possible to resolve the resource consumption
> > > > > problem
> > > > > > > of Pulsar PR builds in the way that Ali is recommending, without
> > > > > switching
> > > > > > > from multiple workflows to a single workflow that can reuse binary
> > > > > > > artifacts in the build.
> > > > > > >
> > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > > more
> > > > > > > > on solving flaky tests.
> > > > > > > > Consider merging them into one workflow when the tests are in a
> > > > > better
> > > > > > > > situation.
> > > > > > >
> > > > > > > Makes sense for minimizing the risk of change, but we cannot just wait
> > > > > for
> > > > > > > things to fix themselves.
> > > > > > > How long will other Apache projects tolerate the resource consumption
> > > > > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> > > > > For
> > > > > > > example,
> > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > .
> > > > > > > Isn't it urgent to resolve it?
> > > > > > >
> > > > > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> > > > > CI"
> > > > > > > based on the community feedback in the upcoming days. That might mean
> > > > > that
> > > > > > > the current solution is pivoted. The goal is to solve the problems of
> > > > > high
> > > > > > > resource consumption and long lead time for PR build in Pulsar CI.
> > > > > Please
> > > > > > > continue to provide feedback so that we get a revisited plan together
> > > > > that
> > > > > > > addresses these problems.Thank you!
> > > > > > >
> > > > > > > BR,
> > > > > > > -Lari
> > > > > > >
> > > > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > > > >
> > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > > GitHub
> > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > >
> > > > > > > > I see what you referred to before now. This can be easily done. I
> > > > > (or any
> > > > > > > > other committer) can do it for you.
> > > > > > > >
> > > > > > > > There is no real blocker for you to continue work even there are
> > > > > concerns
> > > > > > > > or it is not listed in the PIP.
> > > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > > objection,
> > > > > > > > please go ahead and not need to wait for approval.
> > > > > > > > If people raise concerns, please address the concerns.
> > > > > > > >
> > > > > > > > > The reason why
> > > > > > > > re-runs happen currently is because of the high flakiness of tests.
> > > > > > > >
> > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > > workflows
> > > > > > > > into one giant workflow.
> > > > > > > > It is not about "No pain, no gain". The community has suffered a lot
> > > > > with
> > > > > > > > giant workflow before.
> > > > > > > > There were a lot of committers and contributors working hard to
> > > > > split one
> > > > > > > > giant workflow into multiple
> > > > > > > > current workflows. Unless there is really strong evidence that
> > > > > merging
> > > > > > > them
> > > > > > > > back to one will improve
> > > > > > > > the entire CI experience, I will still have concerns about one giant
> > > > > > > > workflow approach.
> > > > > > > >
> > > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > > more
> > > > > > > > on solving flaky tests.
> > > > > > > > Consider merging them into one workflow when the tests are in a
> > > > > better
> > > > > > > > situation.
> > > > > > > >
> > > > > > > > > This solution would also require disabling
> > > > > > > > required status checks
> > > > > > > >
> > > > > > > > I don't think it is a good idea to disable status checks. We can
> > > > > consider
> > > > > > > > running "dark mode" but it will just overload the action quota.
> > > > > > > >
> > > > > > > > Another alternative is to mirror the pull requests into another
> > > > > Github
> > > > > > > > account to test that and get more concrete statistics on the
> > > > > flakiness
> > > > > > > rate
> > > > > > > > of one workflow approach.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > > > >
> > > > > > > > > Thanks for the feedback, Sijie.
> > > > > > > > >
> > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > > However, I am not sure how much it will help if you run all
> > > > > workflows
> > > > > > > > in
> > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > >
> > > > > > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > > > > > Addressing flakiness continues to be top-priority. Now that the
> > > > > Pulsar
> > > > > > > CI
> > > > > > > > > workflow prototype is finished, I'll be focusing more in the other
> > > > > > > draft
> > > > > > > > > PIP, "Changes to flaky test handling".
> > > > > > > > > We as a community should address the critical problem that the
> > > > > current
> > > > > > > > > retry solution has: it can mask bugs in production code and make
> > > > > the
> > > > > > > > build
> > > > > > > > > pass and allow changes to be merged that cause regressions.
> > > > > > > > > It's a false sense of security what the green builds after all the
> > > > > > > > retries
> > > > > > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > > > > > addressing this.
> > > > > > > > >
> > > > > > > > > If something doesn't work, it can be adapted and improved. Changes
> > > > > can
> > > > > > > be
> > > > > > > > > rolled back and revisited when things go worse. We also need a
> > > > > leap of
> > > > > > > > > faith.
> > > > > > > > > "No pain, no gain", like any change, it will be painful at first,
> > > > > but
> > > > > > > we
> > > > > > > > > will get over the bump.
> > > > > > > > >
> > > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > > not a
> > > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > > approach
> > > > > > > > will
> > > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > > concurrent
> > > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > > bring
> > > > > > > side
> > > > > > > > > > effects?
> > > > > > > > >
> > > > > > > > > One possibility to address this is to introduce the new workflow
> > > > > in a
> > > > > > > > mode
> > > > > > > > > where you need to opt-in to the new workflow in some way.
> > > > > > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > > > > > It might be possible to configure the existing workflow and this
> > > > > new
> > > > > > > > > workflow in a way where some condition (for example whitelisted
> > > > > github
> > > > > > > > user
> > > > > > > > > name or a certain keyword in the PR title/description) chooses
> > > > > either
> > > > > > > one
> > > > > > > > > for the Pull request. This solution would also require disabling
> > > > > > > > > required status checks ("Require status checks to pass before
> > > > > merging"
> > > > > > > > > feature in GitHub branch protection rules) since the names of the
> > > > > > > checks
> > > > > > > > > would be different.
> > > > > > > > >
> > > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > > without
> > > > > > > > > > merging them into one giant workflow?
> > > > > > > > >
> > > > > > > > > This is what I have been doing. All individual optimizations have
> > > > > > > already
> > > > > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > > > > > have
> > > > > > > > > already been merged.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > > > > There's also 2 build related PRs from yesterday which haven't been
> > > > > > > merged
> > > > > > > > > yet:
> > > > > > > > > Fix Maven download issues (ported from the prototype to our
> > > > > existing
> > > > > > > > Pulsar
> > > > > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > > > > improve Maven module build order (required for more efficient
> > > > > builds
> > > > > > > that
> > > > > > > > > selectively build required artifacts):
> > > > > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > > > > >
> > > > > > > > > There aren't many optimizations left that could be ported from the
> > > > > > > > > prototype to the existing workflow. There are a few, but the
> > > > > impact is
> > > > > > > > > minor.
> > > > > > > > > The reason for this is that the optimization with the greatest
> > > > > impact
> > > > > > > are
> > > > > > > > > the ones that build a binary artifacts (maven libs, docker images)
> > > > > once
> > > > > > > > and
> > > > > > > > > share it with the downstream jobs in the pipeline.
> > > > > > > > > Applying this type of solution has certain challenges when there
> > > > > are
> > > > > > > > > multiple separate workflow. Sharing binary artifacts to other
> > > > > workflows
> > > > > > > > > would require that the workflow to reuse the artifacts gets
> > > > > triggered
> > > > > > > by
> > > > > > > > > the workflow that produced the artifacts. This wouldn't be secure
> > > > > or
> > > > > > > > > practical for handling pull requests. Triggering a workflow
> > > > > explicitly
> > > > > > > > > would require a token from the main repository and using that for
> > > > > pull
> > > > > > > > > request builds would be a serious security vulnerability. (more
> > > > > details
> > > > > > > > > about the GitHub Actions security model in
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > > > > )
> > > > > > > > >
> > > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > > GitHub
> > > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> > > > > wiki
> > > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > > >
> > > > > > > > > -Lari
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > This is good progress. However, my main concern is still merging
> > > > > all
> > > > > > > > > > workflows into one giant workflow.
> > > > > > > > > >
> > > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > > However, I am not sure how much it will help if you run all
> > > > > workflows
> > > > > > > > in
> > > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > > >
> > > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > > not a
> > > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > > approach
> > > > > > > > will
> > > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > > concurrent
> > > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > > bring
> > > > > > > side
> > > > > > > > > > effects?
> > > > > > > > > >
> > > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > > without
> > > > > > > > > > merging them into one giant workflow?
> > > > > > > > > >
> > > > > > > > > > - Sijie
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for the feedback Michael.
> > > > > > > > > > >
> > > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > > affect the
> > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > >
> > > > > > > > > > > This is a good question. There isn't a clear answer in the
> > > > > GitHub
> > > > > > > > > Actions
> > > > > > > > > > > Cache documentation.
> > > > > > > > > > >
> > > > > > > > > > > The documentation is
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > > > > .
> > > > > > > > > > > Based on this document and some testing, I have made these
> > > > > > > > conclusions:
> > > > > > > > > > > For GitHub Actions Cache, pull requests get executed in the
> > > > > context
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > forked repository.
> > > > > > > > > > > The workflow triggered by a pull request event can only update
> > > > > it's
> > > > > > > > own
> > > > > > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > > > > > If there's a cache miss, the entry will get written to the
> > > > > cache of
> > > > > > > > the
> > > > > > > > > > > forked repository. If the PR could to write to the upstream
> > > > > cache,
> > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > > be a security issue since this would be vulnerable to cache
> > > > > > > poisoning
> > > > > > > > > > > attacks. Each repository has a 5GB quota for writes. The
> > > > > entries
> > > > > > > are
> > > > > > > > > kept
> > > > > > > > > > > up to 7 days.
> > > > > > > > > > > The performance is fairly good. Loading docker images from the
> > > > > > > > > repository
> > > > > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> > > > > (the
> > > > > > > > > > > performance of the GHA cache is most likely higher since this
> > > > > is
> > > > > > > the
> > > > > > > > > > > throughput for docker load / docker save)
> > > > > > > > > > >
> > > > > > > > > > > If a single repository has a lot of concurrent jobs, it could
> > > > > start
> > > > > > > > > > > evicting caches.
> > > > > > > > > > > However that isn't likely to happen with the way Pulsar is
> > > > > > > developed
> > > > > > > > > > since
> > > > > > > > > > > pull requests are created from personal forks.
> > > > > > > > > > >
> > > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > > reference
> > > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > > assuming
> > > > > > > > > > > that
> > > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > > completion,
> > > > > > > > > > and
> > > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > > I
> > > > > > > > > haven't
> > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > >
> > > > > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> > > > > jobs
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > matrix if one fails. Other parallel flows would run to
> > > > > completion
> > > > > > > by
> > > > > > > > > > > default.
> > > > > > > > > > > In the prototype, I have added a Github script step to each
> > > > > job to
> > > > > > > > > cancel
> > > > > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > > > > Here's an example:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > > > > >
> > > > > > > > > > > The prototype follows a "fail fast" design. When a failure
> > > > > occurs,
> > > > > > > > fail
> > > > > > > > > > > fast and don't continue with other jobs.
> > > > > > > > > > > The benefit of this is that it reduces resource consumption.
> > > > > This
> > > > > > > > helps
> > > > > > > > > > > keep the build queue short.
> > > > > > > > > > > When the build queue is short, developers get quick feedback
> > > > > from
> > > > > > > CI.
> > > > > > > > > > >
> > > > > > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > > > > > *I'm hoping to start a separate document on low level details
> > > > > when
> > > > > > > > > there
> > > > > > > > > > is
> > > > > > > > > > > a high level acceptance of the proposed "Changes to GitHub
> > > > > Actions
> > > > > > > > > based
> > > > > > > > > > > Pulsar CI".*
> > > > > > > > > > > Together we can make this happen. We need decisions too. This
> > > > > > > > proposal
> > > > > > > > > > > cannot stay as a draft forever.
> > > > > > > > > > > I'm looking forward to hearing from the Pulsar community,
> > > > > Pulsar
> > > > > > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > > > > > >
> > > > > > > > > > > BR,
> > > > > > > > > > > -Lari
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > > > > > mikemarsh17@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > > > > > overall,
> > > > > > > > > > it
> > > > > > > > > > > > looks good to me.
> > > > > > > > > > > >
> > > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > > affect the
> > > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > > >
> > > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > > reference
> > > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > > assuming
> > > > > > > > > > > that
> > > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > > completion,
> > > > > > > > > > and
> > > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > > I
> > > > > > > > > haven't
> > > > > > > > > > > > worked with github actions before.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> > > > > community
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > > > > lari.hotari@sagire.fi
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> > > > > has
> > > > > > > > gone
> > > > > > > > > > > > forward
> > > > > > > > > > > > > > based on your feedback. Here are some updates about the
> > > > > work.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > There's a *detailed status update in the document about a
> > > > > > > > > prototype
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> > > > > lot of
> > > > > > > > > > > > improvements
> > > > > > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > > > > > technical
> > > > > > > > > > > > > > obstacles.
> > > > > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> > > > > docker
> > > > > > > > > > images.
> > > > > > > > > > > A
> > > > > > > > > > > > > lot
> > > > > > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > > > > > build
> > > > > > > > to
> > > > > > > > > > > > enable
> > > > > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> > > > > for all
> > > > > > > > PR
> > > > > > > > > > > > reviews
> > > > > > > > > > > > > > and feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> > > > > Pulsar
> > > > > > > > CI"
> > > > > > > > > > work
> > > > > > > > > > > > has
> > > > > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > > > > > build
> > > > > > > > > and
> > > > > > > > > > > to
> > > > > > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > > > > > productivity*
> > > > > > > > > > > > when
> > > > > > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> > > > > feedback.
> > > > > > > > The
> > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > > > > > resource
> > > > > > > > > > > > > consumption
> > > > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > > > > > observed
> > > > > > > > > > > when
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > work began.
> > > > > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> > > > > build,
> > > > > > > the
> > > > > > > > > > > resource
> > > > > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > > > > *This is about 60% reduction in resource consumption.*
> > > > > The
> > > > > > > > whole
> > > > > > > > > > > > pipeline
> > > > > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Here's a breakdown of the duration (resource
> > > > > consumption) of
> > > > > > > > each
> > > > > > > > > > > build
> > > > > > > > > > > > > job
> > > > > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > > > > > 0:06:45
> > > > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> > > > > 0:16:08
> > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> > > > > 0:37:03
> > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > > > > > 0:14:09
> > > > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > > > > > 0:29:55
> > > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > > > > > 0:38:32
> > > > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > > > > 7:08:57
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > However, this is not a showstopper since there are ways
> > > > > to
> > > > > > > > > address
> > > > > > > > > > > the
> > > > > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > > > > There is a separate PIP for changing the way to handle
> > > > > flaky
> > > > > > > > > tests.
> > > > > > > > > > > You
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> > > > > based
> > > > > > > > > > Pulsar
> > > > > > > > > > > > CI"
> > > > > > > > > > > > > > document's header.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > > . *It also contains more details of the prototype that
> > > > > has
> > > > > > > been
> > > > > > > > > > > > > > successfully completed.*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2) *Please share your feedback and suggest a way
> > > > > forward.*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > > > > consuming
> > > > > > > > > the
> > > > > > > > > > > > > majority
> > > > > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > > > > github.com/apache
> > > > > > > > > > > > > > projects.
> > > > > > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > > > > > demand
> > > > > > > > > to
> > > > > > > > > > > > > improve
> > > > > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > > > > <
> > > > > > > > > >
> > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > > > >
> > > > > > > > > > > > > > asap.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > > > > > time
> > > > > > > > > > that a
> > > > > > > > > > > > > > Runner
> > > > > > > > > > > > > > > is occupied. I observed the workflow runs for handling
> > > > > a
> > > > > > > > single
> > > > > > > > > > > Pull
> > > > > > > > > > > > > > > Request (in my personal fork) and these were the
> > > > > running
> > > > > > > > > > durations:
> > > > > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > > > > > Actions
> > > > > > > > > > > > Runners
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > > > > request to
> > > > > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > > > > is
> > > > > > > very
> > > > > > > > > > high,
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > leads to the build queue to grow and take a long time
> > > > > to
> > > > > > > > > process.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > > > > for
> > > > > > > the
> > > > > > > > > > last
> > > > > > > > > > > 3
> > > > > > > > > > > > > > > months. During this period I worked on a few
> > > > > experiments.
> > > > > > > The
> > > > > > > > > > > > learnings
> > > > > > > > > > > > > > > from the past experiments are documented at a high
> > > > > level in
> > > > > > > > the
> > > > > > > > > > > > > following
> > > > > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > > > > CI"
> > > > > > > > > > document
> > > > > > > > > > > > is
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > Google doc:*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > > > > based
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > > > > > > feedback asap. If there's already a similar effort
> > > > > > > ongoing, I
> > > > > > > > > > hope
> > > > > > > > > > > we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > join efforts.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <lh...@apache.org>.
The PR has sufficient reviews, and I'll proceed with merging it today or tomorrow.
Please provide feedback now if you want to do that before the PR is merged.
Thanks!
-Lari
On 2022/03/28 20:05:14 Lari Hotari wrote:
> The PR https://github.com/apache/pulsar/pull/14819 is now ready for review.
>
> The changes in the PR now use GitHub Actions Artifacts for sharing binary files (such as docker images) between the build steps. This saves a lot of GitHub Actions VM resources since the docker images are built once and shared in downstream jobs.
> GitHub Actions Artifacts are meant to be used for sharing data between the jobs in a GitHub Actions workflow [1].
>
> I'm looking forward to your review and feedback on https://github.com/apache/pulsar/pull/14819 .
>
> BR,
>
> -Lari
>
> References:
> [1] GitHub Actions: Storing workflow data as artifacts - https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
>
> On 2022/03/23 10:46:10 Lari Hotari wrote:
> > I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
> >
> > BR,
> > -Lari
> >
> > On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> > > Lari,
> > >
> > > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
> > >
> > > > I have resumed work to improve our GitHub Actions based Pulsar CI.
> > > >
> > > > Last year, I worked on a proof-of-concept which significantly reduced the
> > > > resource consumption and improved the usability of the build by combining
> > > > multiple workflows into a single larger workflow.
> > > >
> > > > The showstopper a year ago was the lack of being able to re-run a single
> > > > failed job in a larger workflow.
> > > > GitHub has since then delivered this feature and no showstoppers are
> > > > present.
> > > >
> > > > I have been posting updates to
> > > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> > > > about the progress.
> > > > I have rebased the changes from last year's PoC, and I'm finalizing and
> > > > testing the changes in my fork under
> > > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> > > > apache/pulsar, when the refactoring is ready.
> > > >
> > >
> > > This is great news !
> > >
> > > Looking forward to your patch
> > >
> > > Enrico
> > >
> > >
> > >
> > > > -Lari
> > > >
> > > > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > > > The prototype has demonstrated about 60% reduction in
> > > > > resource consumption.
> > > > >
> > > > > It is hard to quantify. Merging them into one large workflow can result
> > > > in
> > > > > more failures. Re-running those failures can consume resources as well.
> > > > >
> > > > > > Isn't it urgent to resolve it?
> > > > >
> > > > > I think we are in a stage that gives us breathing room to fix flaky tests
> > > > > and solve other problems, no?
> > > > > I don't mean we stop the effort here. I mean we have other enhancements
> > > > > that we can do to improve the situation.
> > > > > Once we get into a position where the flakiness is reduced, we can merge
> > > > > them into one workflow.
> > > > >
> > > > > - Sijie
> > > > >
> > > > >
> > > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> > > > >
> > > > > > Thanks for the feedback Sijie.
> > > > > >
> > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > objection,
> > > > > > > please go ahead and not need to wait for approval.
> > > > > > > If people raise concerns, please address the concerns.
> > > > > >
> > > > > > You and Ali have raised concerns about changing the existing GitHub
> > > > Actions
> > > > > > workflows in a way where multiple workflows would be combined to a
> > > > single
> > > > > > workflow. Before proceeding, there is a need to address the concerns.
> > > > We
> > > > > > might end up with a completely different type of solution of what has
> > > > been
> > > > > > proposed initially. :)
> > > > > >
> > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > workflows
> > > > > > > into one giant workflow.
> > > > > >
> > > > > > I agree that addressing flaky tests is favorable. The main reason for
> > > > PIP
> > > > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > > > > 2) Reduce lead times for Pull Request feedback
> > > > > > We cannot ignore these problems. If we don't change anything, the
> > > > problems
> > > > > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > > > > resource consumption. Measuring the lead times hasn't been done in the
> > > > > > prototype, but since the reason for long lead times has been long build
> > > > > > queues due to excessive resource consumption, it's likely that the lead
> > > > > > times would be reduced.
> > > > > >
> > > > > > I know that switching to a single workflow isn't the only solution to
> > > > the
> > > > > > above problems. I had a discussion with Ali. He recommended reducing
> > > > the
> > > > > > modules in Pulsar repository (PIP-62), reducing the docker container
> > > > size
> > > > > > and improving the Pulsar Broker unit test harness so that tests using
> > > > it
> > > > > > would be less flaky and that it would be easier to fix the issues in
> > > > > > failing test when there would be better information about what was the
> > > > > > state problem that caused the test to fail.
> > > > > >
> > > > > > As mentioned in the earlier email about the optimizations in the
> > > > Pulsar CI
> > > > > > refactoring prototype, the main benefits come from reusing binary
> > > > artifacts
> > > > > > from previous build stages so that each job doesn't have to build
> > > > > > everything from scratch. This becomes irrelevant when the build is very
> > > > > > fast and there isn't a benefit of reusing artifacts.
> > > > > > This means that it's possible to resolve the resource consumption
> > > > problem
> > > > > > of Pulsar PR builds in the way that Ali is recommending, without
> > > > switching
> > > > > > from multiple workflows to a single workflow that can reuse binary
> > > > > > artifacts in the build.
> > > > > >
> > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > more
> > > > > > > on solving flaky tests.
> > > > > > > Consider merging them into one workflow when the tests are in a
> > > > better
> > > > > > > situation.
> > > > > >
> > > > > > Makes sense for minimizing the risk of change, but we cannot just wait
> > > > for
> > > > > > things to fix themselves.
> > > > > > How long will other Apache projects tolerate the resource consumption
> > > > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> > > > For
> > > > > > example,
> > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > .
> > > > > > Isn't it urgent to resolve it?
> > > > > >
> > > > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> > > > CI"
> > > > > > based on the community feedback in the upcoming days. That might mean
> > > > that
> > > > > > the current solution is pivoted. The goal is to solve the problems of
> > > > high
> > > > > > resource consumption and long lead time for PR build in Pulsar CI.
> > > > Please
> > > > > > continue to provide feedback so that we get a revisited plan together
> > > > that
> > > > > > addresses these problems.Thank you!
> > > > > >
> > > > > > BR,
> > > > > > -Lari
> > > > > >
> > > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > > >
> > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > GitHub
> > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > >
> > > > > > > I see what you referred to before now. This can be easily done. I
> > > > (or any
> > > > > > > other committer) can do it for you.
> > > > > > >
> > > > > > > There is no real blocker for you to continue work even there are
> > > > concerns
> > > > > > > or it is not listed in the PIP.
> > > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > > objection,
> > > > > > > please go ahead and not need to wait for approval.
> > > > > > > If people raise concerns, please address the concerns.
> > > > > > >
> > > > > > > > The reason why
> > > > > > > re-runs happen currently is because of the high flakiness of tests.
> > > > > > >
> > > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > > workflows
> > > > > > > into one giant workflow.
> > > > > > > It is not about "No pain, no gain". The community has suffered a lot
> > > > with
> > > > > > > giant workflow before.
> > > > > > > There were a lot of committers and contributors working hard to
> > > > split one
> > > > > > > giant workflow into multiple
> > > > > > > current workflows. Unless there is really strong evidence that
> > > > merging
> > > > > > them
> > > > > > > back to one will improve
> > > > > > > the entire CI experience, I will still have concerns about one giant
> > > > > > > workflow approach.
> > > > > > >
> > > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > > more
> > > > > > > on solving flaky tests.
> > > > > > > Consider merging them into one workflow when the tests are in a
> > > > better
> > > > > > > situation.
> > > > > > >
> > > > > > > > This solution would also require disabling
> > > > > > > required status checks
> > > > > > >
> > > > > > > I don't think it is a good idea to disable status checks. We can
> > > > consider
> > > > > > > running "dark mode" but it will just overload the action quota.
> > > > > > >
> > > > > > > Another alternative is to mirror the pull requests into another
> > > > Github
> > > > > > > account to test that and get more concrete statistics on the
> > > > flakiness
> > > > > > rate
> > > > > > > of one workflow approach.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > > >
> > > > > > > > Thanks for the feedback, Sijie.
> > > > > > > >
> > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > there
> > > > > > > > are
> > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > However, I am not sure how much it will help if you run all
> > > > workflows
> > > > > > > in
> > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > >
> > > > > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > > > > Addressing flakiness continues to be top-priority. Now that the
> > > > Pulsar
> > > > > > CI
> > > > > > > > workflow prototype is finished, I'll be focusing more in the other
> > > > > > draft
> > > > > > > > PIP, "Changes to flaky test handling".
> > > > > > > > We as a community should address the critical problem that the
> > > > current
> > > > > > > > retry solution has: it can mask bugs in production code and make
> > > > the
> > > > > > > build
> > > > > > > > pass and allow changes to be merged that cause regressions.
> > > > > > > > It's a false sense of security what the green builds after all the
> > > > > > > retries
> > > > > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > > > > addressing this.
> > > > > > > >
> > > > > > > > If something doesn't work, it can be adapted and improved. Changes
> > > > can
> > > > > > be
> > > > > > > > rolled back and revisited when things go worse. We also need a
> > > > leap of
> > > > > > > > faith.
> > > > > > > > "No pain, no gain", like any change, it will be painful at first,
> > > > but
> > > > > > we
> > > > > > > > will get over the bump.
> > > > > > > >
> > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > not a
> > > > > > > lot
> > > > > > > > of
> > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > approach
> > > > > > > will
> > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > concurrent
> > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > bring
> > > > > > side
> > > > > > > > > effects?
> > > > > > > >
> > > > > > > > One possibility to address this is to introduce the new workflow
> > > > in a
> > > > > > > mode
> > > > > > > > where you need to opt-in to the new workflow in some way.
> > > > > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > > > > It might be possible to configure the existing workflow and this
> > > > new
> > > > > > > > workflow in a way where some condition (for example whitelisted
> > > > github
> > > > > > > user
> > > > > > > > name or a certain keyword in the PR title/description) chooses
> > > > either
> > > > > > one
> > > > > > > > for the Pull request. This solution would also require disabling
> > > > > > > > required status checks ("Require status checks to pass before
> > > > merging"
> > > > > > > > feature in GitHub branch protection rules) since the names of the
> > > > > > checks
> > > > > > > > would be different.
> > > > > > > >
> > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > without
> > > > > > > > > merging them into one giant workflow?
> > > > > > > >
> > > > > > > > This is what I have been doing. All individual optimizations have
> > > > > > already
> > > > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > > > > have
> > > > > > > > already been merged.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > > > There's also 2 build related PRs from yesterday which haven't been
> > > > > > merged
> > > > > > > > yet:
> > > > > > > > Fix Maven download issues (ported from the prototype to our
> > > > existing
> > > > > > > Pulsar
> > > > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > > > improve Maven module build order (required for more efficient
> > > > builds
> > > > > > that
> > > > > > > > selectively build required artifacts):
> > > > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > > > >
> > > > > > > > There aren't many optimizations left that could be ported from the
> > > > > > > > prototype to the existing workflow. There are a few, but the
> > > > impact is
> > > > > > > > minor.
> > > > > > > > The reason for this is that the optimization with the greatest
> > > > impact
> > > > > > are
> > > > > > > > the ones that build a binary artifacts (maven libs, docker images)
> > > > once
> > > > > > > and
> > > > > > > > share it with the downstream jobs in the pipeline.
> > > > > > > > Applying this type of solution has certain challenges when there
> > > > are
> > > > > > > > multiple separate workflow. Sharing binary artifacts to other
> > > > workflows
> > > > > > > > would require that the workflow to reuse the artifacts gets
> > > > triggered
> > > > > > by
> > > > > > > > the workflow that produced the artifacts. This wouldn't be secure
> > > > or
> > > > > > > > practical for handling pull requests. Triggering a workflow
> > > > explicitly
> > > > > > > > would require a token from the main repository and using that for
> > > > pull
> > > > > > > > request builds would be a serious security vulnerability. (more
> > > > details
> > > > > > > > about the GitHub Actions security model in
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > > > )
> > > > > > > >
> > > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > > GitHub
> > > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> > > > wiki
> > > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > > >
> > > > > > > > -Lari
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > This is good progress. However, my main concern is still merging
> > > > all
> > > > > > > > > workflows into one giant workflow.
> > > > > > > > >
> > > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > > there
> > > > > > > > are
> > > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > > However, I am not sure how much it will help if you run all
> > > > workflows
> > > > > > > in
> > > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > > >
> > > > > > > > > Secondly, your test has been done in your folk where there are
> > > > not a
> > > > > > > lot
> > > > > > > > of
> > > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > > approach
> > > > > > > will
> > > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > > concurrent
> > > > > > > > > pull requests in your account to prove your approach doesn't
> > > > bring
> > > > > > side
> > > > > > > > > effects?
> > > > > > > > >
> > > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > > without
> > > > > > > > > merging them into one giant workflow?
> > > > > > > > >
> > > > > > > > > - Sijie
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for the feedback Michael.
> > > > > > > > > >
> > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > affect the
> > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > >
> > > > > > > > > > This is a good question. There isn't a clear answer in the
> > > > GitHub
> > > > > > > > Actions
> > > > > > > > > > Cache documentation.
> > > > > > > > > >
> > > > > > > > > > The documentation is
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > > > .
> > > > > > > > > > Based on this document and some testing, I have made these
> > > > > > > conclusions:
> > > > > > > > > > For GitHub Actions Cache, pull requests get executed in the
> > > > context
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > forked repository.
> > > > > > > > > > The workflow triggered by a pull request event can only update
> > > > it's
> > > > > > > own
> > > > > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > > > > If there's a cache miss, the entry will get written to the
> > > > cache of
> > > > > > > the
> > > > > > > > > > forked repository. If the PR could to write to the upstream
> > > > cache,
> > > > > > it
> > > > > > > > > would
> > > > > > > > > > be a security issue since this would be vulnerable to cache
> > > > > > poisoning
> > > > > > > > > > attacks. Each repository has a 5GB quota for writes. The
> > > > entries
> > > > > > are
> > > > > > > > kept
> > > > > > > > > > up to 7 days.
> > > > > > > > > > The performance is fairly good. Loading docker images from the
> > > > > > > > repository
> > > > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> > > > (the
> > > > > > > > > > performance of the GHA cache is most likely higher since this
> > > > is
> > > > > > the
> > > > > > > > > > throughput for docker load / docker save)
> > > > > > > > > >
> > > > > > > > > > If a single repository has a lot of concurrent jobs, it could
> > > > start
> > > > > > > > > > evicting caches.
> > > > > > > > > > However that isn't likely to happen with the way Pulsar is
> > > > > > developed
> > > > > > > > > since
> > > > > > > > > > pull requests are created from personal forks.
> > > > > > > > > >
> > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > reference
> > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > assuming
> > > > > > > > > > that
> > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > completion,
> > > > > > > > > and
> > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > I
> > > > > > > > haven't
> > > > > > > > > > > worked with github actions before.
> > > > > > > > > >
> > > > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> > > > jobs
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > matrix if one fails. Other parallel flows would run to
> > > > completion
> > > > > > by
> > > > > > > > > > default.
> > > > > > > > > > In the prototype, I have added a Github script step to each
> > > > job to
> > > > > > > > cancel
> > > > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > > > Here's an example:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > > > >
> > > > > > > > > > The prototype follows a "fail fast" design. When a failure
> > > > occurs,
> > > > > > > fail
> > > > > > > > > > fast and don't continue with other jobs.
> > > > > > > > > > The benefit of this is that it reduces resource consumption.
> > > > This
> > > > > > > helps
> > > > > > > > > > keep the build queue short.
> > > > > > > > > > When the build queue is short, developers get quick feedback
> > > > from
> > > > > > CI.
> > > > > > > > > >
> > > > > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > > > > *I'm hoping to start a separate document on low level details
> > > > when
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > a high level acceptance of the proposed "Changes to GitHub
> > > > Actions
> > > > > > > > based
> > > > > > > > > > Pulsar CI".*
> > > > > > > > > > Together we can make this happen. We need decisions too. This
> > > > > > > proposal
> > > > > > > > > > cannot stay as a draft forever.
> > > > > > > > > > I'm looking forward to hearing from the Pulsar community,
> > > > Pulsar
> > > > > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > > > > >
> > > > > > > > > > BR,
> > > > > > > > > > -Lari
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > > > > mikemarsh17@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > > > > overall,
> > > > > > > > > it
> > > > > > > > > > > looks good to me.
> > > > > > > > > > >
> > > > > > > > > > > I left a question on the doc about how concurrent runs
> > > > affect the
> > > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > > >
> > > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > > reference
> > > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > > assuming
> > > > > > > > > > that
> > > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > > completion,
> > > > > > > > > and
> > > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > > I
> > > > > > > > haven't
> > > > > > > > > > > worked with github actions before.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> > > > community
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > > > lari.hotari@sagire.fi
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > >
> > > > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> > > > has
> > > > > > > gone
> > > > > > > > > > > forward
> > > > > > > > > > > > > based on your feedback. Here are some updates about the
> > > > work.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > There's a *detailed status update in the document about a
> > > > > > > > prototype
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> > > > lot of
> > > > > > > > > > > improvements
> > > > > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > > > > technical
> > > > > > > > > > > > > obstacles.
> > > > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> > > > docker
> > > > > > > > > images.
> > > > > > > > > > A
> > > > > > > > > > > > lot
> > > > > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > > > > build
> > > > > > > to
> > > > > > > > > > > enable
> > > > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> > > > for all
> > > > > > > PR
> > > > > > > > > > > reviews
> > > > > > > > > > > > > and feedback.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> > > > Pulsar
> > > > > > > CI"
> > > > > > > > > work
> > > > > > > > > > > has
> > > > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > > > > build
> > > > > > > > and
> > > > > > > > > > to
> > > > > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > > > > productivity*
> > > > > > > > > > > when
> > > > > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> > > > feedback.
> > > > > > > The
> > > > > > > > > > > > prototype
> > > > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > > > >
> > > > > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > > > > resource
> > > > > > > > > > > > consumption
> > > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > > > > observed
> > > > > > > > > > when
> > > > > > > > > > > > the
> > > > > > > > > > > > > work began.
> > > > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> > > > build,
> > > > > > the
> > > > > > > > > > resource
> > > > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > > > *This is about 60% reduction in resource consumption.*
> > > > The
> > > > > > > whole
> > > > > > > > > > > pipeline
> > > > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here's a breakdown of the duration (resource
> > > > consumption) of
> > > > > > > each
> > > > > > > > > > build
> > > > > > > > > > > > job
> > > > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > > > > 0:06:45
> > > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> > > > 0:16:08
> > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> > > > 0:37:03
> > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > > > > 0:14:09
> > > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > > > > 0:29:55
> > > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > > > > 0:38:32
> > > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > > > 7:08:57
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > > > ).
> > > > > > > > > > > > > However, this is not a showstopper since there are ways
> > > > to
> > > > > > > > address
> > > > > > > > > > the
> > > > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > > > There is a separate PIP for changing the way to handle
> > > > flaky
> > > > > > > > tests.
> > > > > > > > > > You
> > > > > > > > > > > > can
> > > > > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> > > > based
> > > > > > > > > Pulsar
> > > > > > > > > > > CI"
> > > > > > > > > > > > > document's header.
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > > . *It also contains more details of the prototype that
> > > > has
> > > > > > been
> > > > > > > > > > > > > successfully completed.*
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2) *Please share your feedback and suggest a way
> > > > forward.*
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > > > >
> > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > > > > lari.hotari@sagire.fi
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > > > consuming
> > > > > > > > the
> > > > > > > > > > > > majority
> > > > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > > > github.com/apache
> > > > > > > > > > > > > projects.
> > > > > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > > > > demand
> > > > > > > > to
> > > > > > > > > > > > improve
> > > > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > > > <
> > > > > > > > >
> > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > > >
> > > > > > > > > > > > > asap.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > > > > time
> > > > > > > > > that a
> > > > > > > > > > > > > Runner
> > > > > > > > > > > > > > is occupied. I observed the workflow runs for handling
> > > > a
> > > > > > > single
> > > > > > > > > > Pull
> > > > > > > > > > > > > > Request (in my personal fork) and these were the
> > > > running
> > > > > > > > > durations:
> > > > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > > > > Actions
> > > > > > > > > > > Runners
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > > > request to
> > > > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > > > is
> > > > > > very
> > > > > > > > > high,
> > > > > > > > > > > > this
> > > > > > > > > > > > > > leads to the build queue to grow and take a long time
> > > > to
> > > > > > > > process.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > > > for
> > > > > > the
> > > > > > > > > last
> > > > > > > > > > 3
> > > > > > > > > > > > > > months. During this period I worked on a few
> > > > experiments.
> > > > > > The
> > > > > > > > > > > learnings
> > > > > > > > > > > > > > from the past experiments are documented at a high
> > > > level in
> > > > > > > the
> > > > > > > > > > > > following
> > > > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > > > CI"
> > > > > > > > > document
> > > > > > > > > > > is
> > > > > > > > > > > > a
> > > > > > > > > > > > > > Google doc:*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > > > based
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > > > > feedback asap. If there's already a similar effort
> > > > > > ongoing, I
> > > > > > > > > hope
> > > > > > > > > > we
> > > > > > > > > > > > can
> > > > > > > > > > > > > > join efforts.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <lh...@apache.org>.
The PR https://github.com/apache/pulsar/pull/14819 is now ready for review.
The changes in the PR now use GitHub Actions Artifacts for sharing binary files (such as docker images) between the build steps. This saves a lot of GitHub Actions VM resources since the docker images are built once and shared in downstream jobs.
GitHub Actions Artifacts are meant to be used for sharing data between the jobs in a GitHub Actions workflow [1].
I'm looking forward to your review and feedback on https://github.com/apache/pulsar/pull/14819 .
BR,
-Lari
References:
[1] GitHub Actions: Storing workflow data as artifacts - https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
On 2022/03/23 10:46:10 Lari Hotari wrote:
> I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
>
> BR,
> -Lari
>
> On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> > Lari,
> >
> > Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
> >
> > > I have resumed work to improve our GitHub Actions based Pulsar CI.
> > >
> > > Last year, I worked on a proof-of-concept which significantly reduced the
> > > resource consumption and improved the usability of the build by combining
> > > multiple workflows into a single larger workflow.
> > >
> > > The showstopper a year ago was the lack of being able to re-run a single
> > > failed job in a larger workflow.
> > > GitHub has since then delivered this feature and no showstoppers are
> > > present.
> > >
> > > I have been posting updates to
> > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> > > about the progress.
> > > I have rebased the changes from last year's PoC, and I'm finalizing and
> > > testing the changes in my fork under
> > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> > > apache/pulsar, when the refactoring is ready.
> > >
> >
> > This is great news !
> >
> > Looking forward to your patch
> >
> > Enrico
> >
> >
> >
> > > -Lari
> > >
> > > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > > The prototype has demonstrated about 60% reduction in
> > > > resource consumption.
> > > >
> > > > It is hard to quantify. Merging them into one large workflow can result
> > > in
> > > > more failures. Re-running those failures can consume resources as well.
> > > >
> > > > > Isn't it urgent to resolve it?
> > > >
> > > > I think we are in a stage that gives us breathing room to fix flaky tests
> > > > and solve other problems, no?
> > > > I don't mean we stop the effort here. I mean we have other enhancements
> > > > that we can do to improve the situation.
> > > > Once we get into a position where the flakiness is reduced, we can merge
> > > > them into one workflow.
> > > >
> > > > - Sijie
> > > >
> > > >
> > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> > > >
> > > > > Thanks for the feedback Sijie.
> > > > >
> > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > objection,
> > > > > > please go ahead and not need to wait for approval.
> > > > > > If people raise concerns, please address the concerns.
> > > > >
> > > > > You and Ali have raised concerns about changing the existing GitHub
> > > Actions
> > > > > workflows in a way where multiple workflows would be combined to a
> > > single
> > > > > workflow. Before proceeding, there is a need to address the concerns.
> > > We
> > > > > might end up with a completely different type of solution of what has
> > > been
> > > > > proposed initially. :)
> > > > >
> > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > workflows
> > > > > > into one giant workflow.
> > > > >
> > > > > I agree that addressing flaky tests is favorable. The main reason for
> > > PIP
> > > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > > > 2) Reduce lead times for Pull Request feedback
> > > > > We cannot ignore these problems. If we don't change anything, the
> > > problems
> > > > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > > > resource consumption. Measuring the lead times hasn't been done in the
> > > > > prototype, but since the reason for long lead times has been long build
> > > > > queues due to excessive resource consumption, it's likely that the lead
> > > > > times would be reduced.
> > > > >
> > > > > I know that switching to a single workflow isn't the only solution to
> > > the
> > > > > above problems. I had a discussion with Ali. He recommended reducing
> > > the
> > > > > modules in Pulsar repository (PIP-62), reducing the docker container
> > > size
> > > > > and improving the Pulsar Broker unit test harness so that tests using
> > > it
> > > > > would be less flaky and that it would be easier to fix the issues in
> > > > > failing test when there would be better information about what was the
> > > > > state problem that caused the test to fail.
> > > > >
> > > > > As mentioned in the earlier email about the optimizations in the
> > > Pulsar CI
> > > > > refactoring prototype, the main benefits come from reusing binary
> > > artifacts
> > > > > from previous build stages so that each job doesn't have to build
> > > > > everything from scratch. This becomes irrelevant when the build is very
> > > > > fast and there isn't a benefit of reusing artifacts.
> > > > > This means that it's possible to resolve the resource consumption
> > > problem
> > > > > of Pulsar PR builds in the way that Ali is recommending, without
> > > switching
> > > > > from multiple workflows to a single workflow that can reuse binary
> > > > > artifacts in the build.
> > > > >
> > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > more
> > > > > > on solving flaky tests.
> > > > > > Consider merging them into one workflow when the tests are in a
> > > better
> > > > > > situation.
> > > > >
> > > > > Makes sense for minimizing the risk of change, but we cannot just wait
> > > for
> > > > > things to fix themselves.
> > > > > How long will other Apache projects tolerate the resource consumption
> > > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> > > For
> > > > > example,
> > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > .
> > > > > Isn't it urgent to resolve it?
> > > > >
> > > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> > > CI"
> > > > > based on the community feedback in the upcoming days. That might mean
> > > that
> > > > > the current solution is pivoted. The goal is to solve the problems of
> > > high
> > > > > resource consumption and long lead time for PR build in Pulsar CI.
> > > Please
> > > > > continue to provide feedback so that we get a revisited plan together
> > > that
> > > > > addresses these problems.Thank you!
> > > > >
> > > > > BR,
> > > > > -Lari
> > > > >
> > > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > > > >
> > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > GitHub
> > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > >
> > > > > > I see what you referred to before now. This can be easily done. I
> > > (or any
> > > > > > other committer) can do it for you.
> > > > > >
> > > > > > There is no real blocker for you to continue work even there are
> > > concerns
> > > > > > or it is not listed in the PIP.
> > > > > > We are using a lazy consensus approach. Typically if there is no
> > > > > objection,
> > > > > > please go ahead and not need to wait for approval.
> > > > > > If people raise concerns, please address the concerns.
> > > > > >
> > > > > > > The reason why
> > > > > > re-runs happen currently is because of the high flakiness of tests.
> > > > > >
> > > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > > workflows
> > > > > > into one giant workflow.
> > > > > > It is not about "No pain, no gain". The community has suffered a lot
> > > with
> > > > > > giant workflow before.
> > > > > > There were a lot of committers and contributors working hard to
> > > split one
> > > > > > giant workflow into multiple
> > > > > > current workflows. Unless there is really strong evidence that
> > > merging
> > > > > them
> > > > > > back to one will improve
> > > > > > the entire CI experience, I will still have concerns about one giant
> > > > > > workflow approach.
> > > > > >
> > > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > > more
> > > > > > on solving flaky tests.
> > > > > > Consider merging them into one workflow when the tests are in a
> > > better
> > > > > > situation.
> > > > > >
> > > > > > > This solution would also require disabling
> > > > > > required status checks
> > > > > >
> > > > > > I don't think it is a good idea to disable status checks. We can
> > > consider
> > > > > > running "dark mode" but it will just overload the action quota.
> > > > > >
> > > > > > Another alternative is to mirror the pull requests into another
> > > Github
> > > > > > account to test that and get more concrete statistics on the
> > > flakiness
> > > > > rate
> > > > > > of one workflow approach.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > > > >
> > > > > > > Thanks for the feedback, Sijie.
> > > > > > >
> > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > there
> > > > > > > are
> > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > However, I am not sure how much it will help if you run all
> > > workflows
> > > > > > in
> > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > >
> > > > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > > > Addressing flakiness continues to be top-priority. Now that the
> > > Pulsar
> > > > > CI
> > > > > > > workflow prototype is finished, I'll be focusing more in the other
> > > > > draft
> > > > > > > PIP, "Changes to flaky test handling".
> > > > > > > We as a community should address the critical problem that the
> > > current
> > > > > > > retry solution has: it can mask bugs in production code and make
> > > the
> > > > > > build
> > > > > > > pass and allow changes to be merged that cause regressions.
> > > > > > > It's a false sense of security what the green builds after all the
> > > > > > retries
> > > > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > > > addressing this.
> > > > > > >
> > > > > > > If something doesn't work, it can be adapted and improved. Changes
> > > can
> > > > > be
> > > > > > > rolled back and revisited when things go worse. We also need a
> > > leap of
> > > > > > > faith.
> > > > > > > "No pain, no gain", like any change, it will be painful at first,
> > > but
> > > > > we
> > > > > > > will get over the bump.
> > > > > > >
> > > > > > > > Secondly, your test has been done in your folk where there are
> > > not a
> > > > > > lot
> > > > > > > of
> > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > approach
> > > > > > will
> > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > concurrent
> > > > > > > > pull requests in your account to prove your approach doesn't
> > > bring
> > > > > side
> > > > > > > > effects?
> > > > > > >
> > > > > > > One possibility to address this is to introduce the new workflow
> > > in a
> > > > > > mode
> > > > > > > where you need to opt-in to the new workflow in some way.
> > > > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > > > It might be possible to configure the existing workflow and this
> > > new
> > > > > > > workflow in a way where some condition (for example whitelisted
> > > github
> > > > > > user
> > > > > > > name or a certain keyword in the PR title/description) chooses
> > > either
> > > > > one
> > > > > > > for the Pull request. This solution would also require disabling
> > > > > > > required status checks ("Require status checks to pass before
> > > merging"
> > > > > > > feature in GitHub branch protection rules) since the names of the
> > > > > checks
> > > > > > > would be different.
> > > > > > >
> > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > without
> > > > > > > > merging them into one giant workflow?
> > > > > > >
> > > > > > > This is what I have been doing. All individual optimizations have
> > > > > already
> > > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > > > have
> > > > > > > already been merged.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > > There's also 2 build related PRs from yesterday which haven't been
> > > > > merged
> > > > > > > yet:
> > > > > > > Fix Maven download issues (ported from the prototype to our
> > > existing
> > > > > > Pulsar
> > > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > > improve Maven module build order (required for more efficient
> > > builds
> > > > > that
> > > > > > > selectively build required artifacts):
> > > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > > >
> > > > > > > There aren't many optimizations left that could be ported from the
> > > > > > > prototype to the existing workflow. There are a few, but the
> > > impact is
> > > > > > > minor.
> > > > > > > The reason for this is that the optimization with the greatest
> > > impact
> > > > > are
> > > > > > > the ones that build a binary artifacts (maven libs, docker images)
> > > once
> > > > > > and
> > > > > > > share it with the downstream jobs in the pipeline.
> > > > > > > Applying this type of solution has certain challenges when there
> > > are
> > > > > > > multiple separate workflow. Sharing binary artifacts to other
> > > workflows
> > > > > > > would require that the workflow to reuse the artifacts gets
> > > triggered
> > > > > by
> > > > > > > the workflow that produced the artifacts. This wouldn't be secure
> > > or
> > > > > > > practical for handling pull requests. Triggering a workflow
> > > explicitly
> > > > > > > would require a token from the main repository and using that for
> > > pull
> > > > > > > request builds would be a serious security vulnerability. (more
> > > details
> > > > > > > about the GitHub Actions security model in
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > > )
> > > > > > >
> > > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > > GitHub
> > > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> > > wiki
> > > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > > >
> > > > > > > -Lari
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > This is good progress. However, my main concern is still merging
> > > all
> > > > > > > > workflows into one giant workflow.
> > > > > > > >
> > > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > > there
> > > > > > > are
> > > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > > However, I am not sure how much it will help if you run all
> > > workflows
> > > > > > in
> > > > > > > > one giant workflow. Or is it making things worse?
> > > > > > > >
> > > > > > > > Secondly, your test has been done in your folk where there are
> > > not a
> > > > > > lot
> > > > > > > of
> > > > > > > > concurrent pushes and pull requests. I am not sure how your
> > > approach
> > > > > > will
> > > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > > concurrent
> > > > > > > > pull requests in your account to prove your approach doesn't
> > > bring
> > > > > side
> > > > > > > > effects?
> > > > > > > >
> > > > > > > > Lastly, can we apply those optimizations to current workflows
> > > without
> > > > > > > > merging them into one giant workflow?
> > > > > > > >
> > > > > > > > - Sijie
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for the feedback Michael.
> > > > > > > > >
> > > > > > > > > > I left a question on the doc about how concurrent runs
> > > affect the
> > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > >
> > > > > > > > > This is a good question. There isn't a clear answer in the
> > > GitHub
> > > > > > > Actions
> > > > > > > > > Cache documentation.
> > > > > > > > >
> > > > > > > > > The documentation is
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > > .
> > > > > > > > > Based on this document and some testing, I have made these
> > > > > > conclusions:
> > > > > > > > > For GitHub Actions Cache, pull requests get executed in the
> > > context
> > > > > > of
> > > > > > > > the
> > > > > > > > > forked repository.
> > > > > > > > > The workflow triggered by a pull request event can only update
> > > it's
> > > > > > own
> > > > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > > > If there's a cache miss, the entry will get written to the
> > > cache of
> > > > > > the
> > > > > > > > > forked repository. If the PR could to write to the upstream
> > > cache,
> > > > > it
> > > > > > > > would
> > > > > > > > > be a security issue since this would be vulnerable to cache
> > > > > poisoning
> > > > > > > > > attacks. Each repository has a 5GB quota for writes. The
> > > entries
> > > > > are
> > > > > > > kept
> > > > > > > > > up to 7 days.
> > > > > > > > > The performance is fairly good. Loading docker images from the
> > > > > > > repository
> > > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> > > (the
> > > > > > > > > performance of the GHA cache is most likely higher since this
> > > is
> > > > > the
> > > > > > > > > throughput for docker load / docker save)
> > > > > > > > >
> > > > > > > > > If a single repository has a lot of concurrent jobs, it could
> > > start
> > > > > > > > > evicting caches.
> > > > > > > > > However that isn't likely to happen with the way Pulsar is
> > > > > developed
> > > > > > > > since
> > > > > > > > > pull requests are created from personal forks.
> > > > > > > > >
> > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > reference
> > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > assuming
> > > > > > > > > that
> > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > completion,
> > > > > > > > and
> > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > I
> > > > > > > haven't
> > > > > > > > > > worked with github actions before.
> > > > > > > > >
> > > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> > > jobs
> > > > > > in
> > > > > > > > the
> > > > > > > > > matrix if one fails. Other parallel flows would run to
> > > completion
> > > > > by
> > > > > > > > > default.
> > > > > > > > > In the prototype, I have added a Github script step to each
> > > job to
> > > > > > > cancel
> > > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > > Here's an example:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > > >
> > > > > > > > > The prototype follows a "fail fast" design. When a failure
> > > occurs,
> > > > > > fail
> > > > > > > > > fast and don't continue with other jobs.
> > > > > > > > > The benefit of this is that it reduces resource consumption.
> > > This
> > > > > > helps
> > > > > > > > > keep the build queue short.
> > > > > > > > > When the build queue is short, developers get quick feedback
> > > from
> > > > > CI.
> > > > > > > > >
> > > > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > > > *I'm hoping to start a separate document on low level details
> > > when
> > > > > > > there
> > > > > > > > is
> > > > > > > > > a high level acceptance of the proposed "Changes to GitHub
> > > Actions
> > > > > > > based
> > > > > > > > > Pulsar CI".*
> > > > > > > > > Together we can make this happen. We need decisions too. This
> > > > > > proposal
> > > > > > > > > cannot stay as a draft forever.
> > > > > > > > > I'm looking forward to hearing from the Pulsar community,
> > > Pulsar
> > > > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > > > >
> > > > > > > > > BR,
> > > > > > > > > -Lari
> > > > > > > > >
> > > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > > > mikemarsh17@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > > > overall,
> > > > > > > > it
> > > > > > > > > > looks good to me.
> > > > > > > > > >
> > > > > > > > > > I left a question on the doc about how concurrent runs
> > > affect the
> > > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > > >
> > > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > > reference
> > > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > > assuming
> > > > > > > > > that
> > > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > > completion,
> > > > > > > > and
> > > > > > > > > > that the rest of the remaining stages will get canceled, but
> > > I
> > > > > > > haven't
> > > > > > > > > > worked with github actions before.
> > > > > > > > > >
> > > > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> > > community
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > > lari.hotari@sagire.fi
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > >
> > > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> > > has
> > > > > > gone
> > > > > > > > > > forward
> > > > > > > > > > > > based on your feedback. Here are some updates about the
> > > work.
> > > > > > > > > > > >
> > > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > There's a *detailed status update in the document about a
> > > > > > > prototype
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> > > lot of
> > > > > > > > > > improvements
> > > > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > > > technical
> > > > > > > > > > > > obstacles.
> > > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> > > docker
> > > > > > > > images.
> > > > > > > > > A
> > > > > > > > > > > lot
> > > > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > > > build
> > > > > > to
> > > > > > > > > > enable
> > > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> > > for all
> > > > > > PR
> > > > > > > > > > reviews
> > > > > > > > > > > > and feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> > > Pulsar
> > > > > > CI"
> > > > > > > > work
> > > > > > > > > > has
> > > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > > > build
> > > > > > > and
> > > > > > > > > to
> > > > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > > > productivity*
> > > > > > > > > > when
> > > > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> > > feedback.
> > > > > > The
> > > > > > > > > > > prototype
> > > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > > >
> > > > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > > > resource
> > > > > > > > > > > consumption
> > > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > > > observed
> > > > > > > > > when
> > > > > > > > > > > the
> > > > > > > > > > > > work began.
> > > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> > > build,
> > > > > the
> > > > > > > > > resource
> > > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > > *This is about 60% reduction in resource consumption.*
> > > The
> > > > > > whole
> > > > > > > > > > pipeline
> > > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > > >
> > > > > > > > > > > > Here's a breakdown of the duration (resource
> > > consumption) of
> > > > > > each
> > > > > > > > > build
> > > > > > > > > > > job
> > > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > > > 0:06:45
> > > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> > > 0:16:08
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> > > 0:37:03
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > > > 0:14:09
> > > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > > > 0:29:55
> > > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > > > 0:38:32
> > > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > > 7:08:57
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > > ).
> > > > > > > > > > > > However, this is not a showstopper since there are ways
> > > to
> > > > > > > address
> > > > > > > > > the
> > > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > > There is a separate PIP for changing the way to handle
> > > flaky
> > > > > > > tests.
> > > > > > > > > You
> > > > > > > > > > > can
> > > > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> > > based
> > > > > > > > Pulsar
> > > > > > > > > > CI"
> > > > > > > > > > > > document's header.
> > > > > > > > > > > >
> > > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > > >
> > > > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > > . *It also contains more details of the prototype that
> > > has
> > > > > been
> > > > > > > > > > > > successfully completed.*
> > > > > > > > > > > >
> > > > > > > > > > > > 2) *Please share your feedback and suggest a way
> > > forward.*
> > > > > > > > > > > >
> > > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > > >
> > > > > > > > > > > > BR, Lari
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > > > lari.hotari@sagire.fi
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > > consuming
> > > > > > > the
> > > > > > > > > > > majority
> > > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > > github.com/apache
> > > > > > > > > > > > projects.
> > > > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > > > demand
> > > > > > > to
> > > > > > > > > > > improve
> > > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > > <
> > > > > > > >
> > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > > >
> > > > > > > > > > > > asap.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > > > time
> > > > > > > > that a
> > > > > > > > > > > > Runner
> > > > > > > > > > > > > is occupied. I observed the workflow runs for handling
> > > a
> > > > > > single
> > > > > > > > > Pull
> > > > > > > > > > > > > Request (in my personal fork) and these were the
> > > running
> > > > > > > > durations:
> > > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > > >
> > > > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > > > Actions
> > > > > > > > > > Runners
> > > > > > > > > > > > is
> > > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > > request to
> > > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > > is
> > > > > very
> > > > > > > > high,
> > > > > > > > > > > this
> > > > > > > > > > > > > leads to the build queue to grow and take a long time
> > > to
> > > > > > > process.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > > for
> > > > > the
> > > > > > > > last
> > > > > > > > > 3
> > > > > > > > > > > > > months. During this period I worked on a few
> > > experiments.
> > > > > The
> > > > > > > > > > learnings
> > > > > > > > > > > > > from the past experiments are documented at a high
> > > level in
> > > > > > the
> > > > > > > > > > > following
> > > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > > >
> > > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > > CI"
> > > > > > > > document
> > > > > > > > > > is
> > > > > > > > > > > a
> > > > > > > > > > > > > Google doc:*
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > > based
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > > feedback asap. If there's already a similar effort
> > > > > ongoing, I
> > > > > > > > hope
> > > > > > > > > we
> > > > > > > > > > > can
> > > > > > > > > > > > > join efforts.
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > > >
> > > > > > > > > > > > > BR, Lari
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <lh...@apache.org>.
I have submitted the PR for refactoring the apache/pulsar GitHub Actions based CI. Please review https://github.com/apache/pulsar/pull/14819 .
BR,
-Lari
On 2022/03/22 13:38:36 Enrico Olivelli wrote:
> Lari,
>
> Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
>
> > I have resumed work to improve our GitHub Actions based Pulsar CI.
> >
> > Last year, I worked on a proof-of-concept which significantly reduced the
> > resource consumption and improved the usability of the build by combining
> > multiple workflows into a single larger workflow.
> >
> > The showstopper a year ago was the lack of being able to re-run a single
> > failed job in a larger workflow.
> > GitHub has since then delivered this feature and no showstoppers are
> > present.
> >
> > I have been posting updates to
> > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> > about the progress.
> > I have rebased the changes from last year's PoC, and I'm finalizing and
> > testing the changes in my fork under
> > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> > apache/pulsar, when the refactoring is ready.
> >
>
> This is great news !
>
> Looking forward to your patch
>
> Enrico
>
>
>
> > -Lari
> >
> > On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > > The prototype has demonstrated about 60% reduction in
> > > resource consumption.
> > >
> > > It is hard to quantify. Merging them into one large workflow can result
> > in
> > > more failures. Re-running those failures can consume resources as well.
> > >
> > > > Isn't it urgent to resolve it?
> > >
> > > I think we are in a stage that gives us breathing room to fix flaky tests
> > > and solve other problems, no?
> > > I don't mean we stop the effort here. I mean we have other enhancements
> > > that we can do to improve the situation.
> > > Once we get into a position where the flakiness is reduced, we can merge
> > > them into one workflow.
> > >
> > > - Sijie
> > >
> > >
> > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> > >
> > > > Thanks for the feedback Sijie.
> > > >
> > > > > We are using a lazy consensus approach. Typically if there is no
> > > > objection,
> > > > > please go ahead and not need to wait for approval.
> > > > > If people raise concerns, please address the concerns.
> > > >
> > > > You and Ali have raised concerns about changing the existing GitHub
> > Actions
> > > > workflows in a way where multiple workflows would be combined to a
> > single
> > > > workflow. Before proceeding, there is a need to address the concerns.
> > We
> > > > might end up with a completely different type of solution of what has
> > been
> > > > proposed initially. :)
> > > >
> > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > workflows
> > > > > into one giant workflow.
> > > >
> > > > I agree that addressing flaky tests is favorable. The main reason for
> > PIP
> > > > "Changes to GitHub Actions based Pulsar CI" is to
> > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > > 2) Reduce lead times for Pull Request feedback
> > > > We cannot ignore these problems. If we don't change anything, the
> > problems
> > > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > > resource consumption. Measuring the lead times hasn't been done in the
> > > > prototype, but since the reason for long lead times has been long build
> > > > queues due to excessive resource consumption, it's likely that the lead
> > > > times would be reduced.
> > > >
> > > > I know that switching to a single workflow isn't the only solution to
> > the
> > > > above problems. I had a discussion with Ali. He recommended reducing
> > the
> > > > modules in Pulsar repository (PIP-62), reducing the docker container
> > size
> > > > and improving the Pulsar Broker unit test harness so that tests using
> > it
> > > > would be less flaky and that it would be easier to fix the issues in
> > > > failing test when there would be better information about what was the
> > > > state problem that caused the test to fail.
> > > >
> > > > As mentioned in the earlier email about the optimizations in the
> > Pulsar CI
> > > > refactoring prototype, the main benefits come from reusing binary
> > artifacts
> > > > from previous build stages so that each job doesn't have to build
> > > > everything from scratch. This becomes irrelevant when the build is very
> > > > fast and there isn't a benefit of reusing artifacts.
> > > > This means that it's possible to resolve the resource consumption
> > problem
> > > > of Pulsar PR builds in the way that Ali is recommending, without
> > switching
> > > > from multiple workflows to a single workflow that can reuse binary
> > > > artifacts in the build.
> > > >
> > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > more
> > > > > on solving flaky tests.
> > > > > Consider merging them into one workflow when the tests are in a
> > better
> > > > > situation.
> > > >
> > > > Makes sense for minimizing the risk of change, but we cannot just wait
> > for
> > > > things to fix themselves.
> > > > How long will other Apache projects tolerate the resource consumption
> > > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> > For
> > > > example,
> > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > .
> > > > Isn't it urgent to resolve it?
> > > >
> > > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> > CI"
> > > > based on the community feedback in the upcoming days. That might mean
> > that
> > > > the current solution is pivoted. The goal is to solve the problems of
> > high
> > > > resource consumption and long lead time for PR build in Pulsar CI.
> > Please
> > > > continue to provide feedback so that we get a revisited plan together
> > that
> > > > addresses these problems.Thank you!
> > > >
> > > > BR,
> > > > -Lari
> > > >
> > > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > > >
> > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > GitHub
> > > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > > page https://github.com/apache/pulsar/wiki
> > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > >
> > > > > I see what you referred to before now. This can be easily done. I
> > (or any
> > > > > other committer) can do it for you.
> > > > >
> > > > > There is no real blocker for you to continue work even there are
> > concerns
> > > > > or it is not listed in the PIP.
> > > > > We are using a lazy consensus approach. Typically if there is no
> > > > objection,
> > > > > please go ahead and not need to wait for approval.
> > > > > If people raise concerns, please address the concerns.
> > > > >
> > > > > > The reason why
> > > > > re-runs happen currently is because of the high flakiness of tests.
> > > > >
> > > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > > workflows
> > > > > into one giant workflow.
> > > > > It is not about "No pain, no gain". The community has suffered a lot
> > with
> > > > > giant workflow before.
> > > > > There were a lot of committers and contributors working hard to
> > split one
> > > > > giant workflow into multiple
> > > > > current workflows. Unless there is really strong evidence that
> > merging
> > > > them
> > > > > back to one will improve
> > > > > the entire CI experience, I will still have concerns about one giant
> > > > > workflow approach.
> > > > >
> > > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > > more
> > > > > on solving flaky tests.
> > > > > Consider merging them into one workflow when the tests are in a
> > better
> > > > > situation.
> > > > >
> > > > > > This solution would also require disabling
> > > > > required status checks
> > > > >
> > > > > I don't think it is a good idea to disable status checks. We can
> > consider
> > > > > running "dark mode" but it will just overload the action quota.
> > > > >
> > > > > Another alternative is to mirror the pull requests into another
> > Github
> > > > > account to test that and get more concrete statistics on the
> > flakiness
> > > > rate
> > > > > of one workflow approach.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > > >
> > > > > > Thanks for the feedback, Sijie.
> > > > > >
> > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > there
> > > > > > are
> > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > However, I am not sure how much it will help if you run all
> > workflows
> > > > > in
> > > > > > > one giant workflow. Or is it making things worse?
> > > > > >
> > > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > > Addressing flakiness continues to be top-priority. Now that the
> > Pulsar
> > > > CI
> > > > > > workflow prototype is finished, I'll be focusing more in the other
> > > > draft
> > > > > > PIP, "Changes to flaky test handling".
> > > > > > We as a community should address the critical problem that the
> > current
> > > > > > retry solution has: it can mask bugs in production code and make
> > the
> > > > > build
> > > > > > pass and allow changes to be merged that cause regressions.
> > > > > > It's a false sense of security what the green builds after all the
> > > > > retries
> > > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > > addressing this.
> > > > > >
> > > > > > If something doesn't work, it can be adapted and improved. Changes
> > can
> > > > be
> > > > > > rolled back and revisited when things go worse. We also need a
> > leap of
> > > > > > faith.
> > > > > > "No pain, no gain", like any change, it will be painful at first,
> > but
> > > > we
> > > > > > will get over the bump.
> > > > > >
> > > > > > > Secondly, your test has been done in your folk where there are
> > not a
> > > > > lot
> > > > > > of
> > > > > > > concurrent pushes and pull requests. I am not sure how your
> > approach
> > > > > will
> > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > concurrent
> > > > > > > pull requests in your account to prove your approach doesn't
> > bring
> > > > side
> > > > > > > effects?
> > > > > >
> > > > > > One possibility to address this is to introduce the new workflow
> > in a
> > > > > mode
> > > > > > where you need to opt-in to the new workflow in some way.
> > > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > > It might be possible to configure the existing workflow and this
> > new
> > > > > > workflow in a way where some condition (for example whitelisted
> > github
> > > > > user
> > > > > > name or a certain keyword in the PR title/description) chooses
> > either
> > > > one
> > > > > > for the Pull request. This solution would also require disabling
> > > > > > required status checks ("Require status checks to pass before
> > merging"
> > > > > > feature in GitHub branch protection rules) since the names of the
> > > > checks
> > > > > > would be different.
> > > > > >
> > > > > > > Lastly, can we apply those optimizations to current workflows
> > without
> > > > > > > merging them into one giant workflow?
> > > > > >
> > > > > > This is what I have been doing. All individual optimizations have
> > > > already
> > > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > > have
> > > > > > already been merged.
> > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > > There's also 2 build related PRs from yesterday which haven't been
> > > > merged
> > > > > > yet:
> > > > > > Fix Maven download issues (ported from the prototype to our
> > existing
> > > > > Pulsar
> > > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > > improve Maven module build order (required for more efficient
> > builds
> > > > that
> > > > > > selectively build required artifacts):
> > > > > > https://github.com/apache/pulsar/pull/9882
> > > > > >
> > > > > > There aren't many optimizations left that could be ported from the
> > > > > > prototype to the existing workflow. There are a few, but the
> > impact is
> > > > > > minor.
> > > > > > The reason for this is that the optimization with the greatest
> > impact
> > > > are
> > > > > > the ones that build a binary artifacts (maven libs, docker images)
> > once
> > > > > and
> > > > > > share it with the downstream jobs in the pipeline.
> > > > > > Applying this type of solution has certain challenges when there
> > are
> > > > > > multiple separate workflow. Sharing binary artifacts to other
> > workflows
> > > > > > would require that the workflow to reuse the artifacts gets
> > triggered
> > > > by
> > > > > > the workflow that produced the artifacts. This wouldn't be secure
> > or
> > > > > > practical for handling pull requests. Triggering a workflow
> > explicitly
> > > > > > would require a token from the main repository and using that for
> > pull
> > > > > > request builds would be a serious security vulnerability. (more
> > details
> > > > > > about the GitHub Actions security model in
> > > > > >
> > > > > >
> > > > >
> > > >
> > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > > )
> > > > > >
> > > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> > GitHub
> > > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> > wiki
> > > > > > page https://github.com/apache/pulsar/wiki
> > > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > > >
> > > > > > -Lari
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > This is good progress. However, my main concern is still merging
> > all
> > > > > > > workflows into one giant workflow.
> > > > > > >
> > > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > > there
> > > > > > are
> > > > > > > multiple workflows and each workflow is retryable.
> > > > > > > However, I am not sure how much it will help if you run all
> > workflows
> > > > > in
> > > > > > > one giant workflow. Or is it making things worse?
> > > > > > >
> > > > > > > Secondly, your test has been done in your folk where there are
> > not a
> > > > > lot
> > > > > > of
> > > > > > > concurrent pushes and pull requests. I am not sure how your
> > approach
> > > > > will
> > > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > > concurrent
> > > > > > > pull requests in your account to prove your approach doesn't
> > bring
> > > > side
> > > > > > > effects?
> > > > > > >
> > > > > > > Lastly, can we apply those optimizations to current workflows
> > without
> > > > > > > merging them into one giant workflow?
> > > > > > >
> > > > > > > - Sijie
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks for the feedback Michael.
> > > > > > > >
> > > > > > > > > I left a question on the doc about how concurrent runs
> > affect the
> > > > > > > > > repository's 5 GB cache limit.
> > > > > > > >
> > > > > > > > This is a good question. There isn't a clear answer in the
> > GitHub
> > > > > > Actions
> > > > > > > > Cache documentation.
> > > > > > > >
> > > > > > > > The documentation is
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > > .
> > > > > > > > Based on this document and some testing, I have made these
> > > > > conclusions:
> > > > > > > > For GitHub Actions Cache, pull requests get executed in the
> > context
> > > > > of
> > > > > > > the
> > > > > > > > forked repository.
> > > > > > > > The workflow triggered by a pull request event can only update
> > it's
> > > > > own
> > > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > > If there's a cache miss, the entry will get written to the
> > cache of
> > > > > the
> > > > > > > > forked repository. If the PR could to write to the upstream
> > cache,
> > > > it
> > > > > > > would
> > > > > > > > be a security issue since this would be vulnerable to cache
> > > > poisoning
> > > > > > > > attacks. Each repository has a 5GB quota for writes. The
> > entries
> > > > are
> > > > > > kept
> > > > > > > > up to 7 days.
> > > > > > > > The performance is fairly good. Loading docker images from the
> > > > > > repository
> > > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> > (the
> > > > > > > > performance of the GHA cache is most likely higher since this
> > is
> > > > the
> > > > > > > > throughput for docker load / docker save)
> > > > > > > >
> > > > > > > > If a single repository has a lot of concurrent jobs, it could
> > start
> > > > > > > > evicting caches.
> > > > > > > > However that isn't likely to happen with the way Pulsar is
> > > > developed
> > > > > > > since
> > > > > > > > pull requests are created from personal forks.
> > > > > > > >
> > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > reference
> > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > assuming
> > > > > > > > that
> > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > completion,
> > > > > > > and
> > > > > > > > > that the rest of the remaining stages will get canceled, but
> > I
> > > > > > haven't
> > > > > > > > > worked with github actions before.
> > > > > > > >
> > > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> > jobs
> > > > > in
> > > > > > > the
> > > > > > > > matrix if one fails. Other parallel flows would run to
> > completion
> > > > by
> > > > > > > > default.
> > > > > > > > In the prototype, I have added a Github script step to each
> > job to
> > > > > > cancel
> > > > > > > > the complete workflow when a failure occurs.
> > > > > > > > Here's an example:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > > >
> > > > > > > > The prototype follows a "fail fast" design. When a failure
> > occurs,
> > > > > fail
> > > > > > > > fast and don't continue with other jobs.
> > > > > > > > The benefit of this is that it reduces resource consumption.
> > This
> > > > > helps
> > > > > > > > keep the build queue short.
> > > > > > > > When the build queue is short, developers get quick feedback
> > from
> > > > CI.
> > > > > > > >
> > > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > > *I'm hoping to start a separate document on low level details
> > when
> > > > > > there
> > > > > > > is
> > > > > > > > a high level acceptance of the proposed "Changes to GitHub
> > Actions
> > > > > > based
> > > > > > > > Pulsar CI".*
> > > > > > > > Together we can make this happen. We need decisions too. This
> > > > > proposal
> > > > > > > > cannot stay as a draft forever.
> > > > > > > > I'm looking forward to hearing from the Pulsar community,
> > Pulsar
> > > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > > >
> > > > > > > > BR,
> > > > > > > > -Lari
> > > > > > > >
> > > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > > mikemarsh17@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > > overall,
> > > > > > > it
> > > > > > > > > looks good to me.
> > > > > > > > >
> > > > > > > > > I left a question on the doc about how concurrent runs
> > affect the
> > > > > > > > > repository's 5 GB cache limit.
> > > > > > > > >
> > > > > > > > > I also think it could be helpful to explicitly document, or
> > > > > reference
> > > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > > assuming
> > > > > > > > that
> > > > > > > > > if an action fails, its parallel peer actions will run to
> > > > > completion,
> > > > > > > and
> > > > > > > > > that the rest of the remaining stages will get canceled, but
> > I
> > > > > > haven't
> > > > > > > > > worked with github actions before.
> > > > > > > > >
> > > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > > >
> > > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> > community
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > > lari.hotari@sagire.fi
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > >
> > > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> > has
> > > > > gone
> > > > > > > > > forward
> > > > > > > > > > > based on your feedback. Here are some updates about the
> > work.
> > > > > > > > > > >
> > > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > There's a *detailed status update in the document about a
> > > > > > prototype
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> > lot of
> > > > > > > > > improvements
> > > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > > technical
> > > > > > > > > > > obstacles.
> > > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> > docker
> > > > > > > images.
> > > > > > > > A
> > > > > > > > > > lot
> > > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > > build
> > > > > to
> > > > > > > > > enable
> > > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> > for all
> > > > > PR
> > > > > > > > > reviews
> > > > > > > > > > > and feedback.
> > > > > > > > > > >
> > > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> > Pulsar
> > > > > CI"
> > > > > > > work
> > > > > > > > > has
> > > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > > build
> > > > > > and
> > > > > > > > to
> > > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > > productivity*
> > > > > > > > > when
> > > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> > feedback.
> > > > > The
> > > > > > > > > > prototype
> > > > > > > > > > > demonstrates these improvements.
> > > > > > > > > > >
> > > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > > resource
> > > > > > > > > > consumption
> > > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > > observed
> > > > > > > > when
> > > > > > > > > > the
> > > > > > > > > > > work began.
> > > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> > build,
> > > > the
> > > > > > > > resource
> > > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > > *This is about 60% reduction in resource consumption.*
> > The
> > > > > whole
> > > > > > > > > pipeline
> > > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > > >
> > > > > > > > > > > Here's a breakdown of the duration (resource
> > consumption) of
> > > > > each
> > > > > > > > build
> > > > > > > > > > job
> > > > > > > > > > > in the refactored workflow:
> > > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > > 0:06:45
> > > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> > 0:16:08
> > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> > 0:37:03
> > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > > 0:14:09
> > > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > > 0:29:55
> > > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > > 0:38:32
> > > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > > *Total resource consumption*
> > > > > > > > > > > 7:08:57
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > > ).
> > > > > > > > > > > However, this is not a showstopper since there are ways
> > to
> > > > > > address
> > > > > > > > the
> > > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > > There is a separate PIP for changing the way to handle
> > flaky
> > > > > > tests.
> > > > > > > > You
> > > > > > > > > > can
> > > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> > based
> > > > > > > Pulsar
> > > > > > > > > CI"
> > > > > > > > > > > document's header.
> > > > > > > > > > >
> > > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > > >
> > > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > > . *It also contains more details of the prototype that
> > has
> > > > been
> > > > > > > > > > > successfully completed.*
> > > > > > > > > > >
> > > > > > > > > > > 2) *Please share your feedback and suggest a way
> > forward.*
> > > > > > > > > > >
> > > > > > > > > > > *Thank you for your help!*
> > > > > > > > > > >
> > > > > > > > > > > BR, Lari
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > > lari.hotari@sagire.fi
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > >
> > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > consuming
> > > > > > the
> > > > > > > > > > majority
> > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > github.com/apache
> > > > > > > > > > > projects.
> > > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > > demand
> > > > > > to
> > > > > > > > > > improve
> > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > <
> > > > > > >
> > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > >
> > > > > > > > > > > asap.
> > > > > > > > > > > >
> > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > > time
> > > > > > > that a
> > > > > > > > > > > Runner
> > > > > > > > > > > > is occupied. I observed the workflow runs for handling
> > a
> > > > > single
> > > > > > > > Pull
> > > > > > > > > > > > Request (in my personal fork) and these were the
> > running
> > > > > > > durations:
> > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > >
> > > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > > Actions
> > > > > > > > > Runners
> > > > > > > > > > > is
> > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > request to
> > > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > >
> > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > is
> > > > very
> > > > > > > high,
> > > > > > > > > > this
> > > > > > > > > > > > leads to the build queue to grow and take a long time
> > to
> > > > > > process.
> > > > > > > > > > > >
> > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > for
> > > > the
> > > > > > > last
> > > > > > > > 3
> > > > > > > > > > > > months. During this period I worked on a few
> > experiments.
> > > > The
> > > > > > > > > learnings
> > > > > > > > > > > > from the past experiments are documented at a high
> > level in
> > > > > the
> > > > > > > > > > following
> > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > >
> > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > CI"
> > > > > > > document
> > > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > > > Google doc:*
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > >
> > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > based
> > > > > on
> > > > > > > the
> > > > > > > > > > > > feedback asap. If there's already a similar effort
> > > > ongoing, I
> > > > > > > hope
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > join efforts.
> > > > > > > > > > > >
> > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > >
> > > > > > > > > > > > BR, Lari
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Enrico Olivelli <eo...@gmail.com>.
Lari,
Il Mar 22 Mar 2022, 14:32 Lari Hotari <lh...@apache.org> ha scritto:
> I have resumed work to improve our GitHub Actions based Pulsar CI.
>
> Last year, I worked on a proof-of-concept which significantly reduced the
> resource consumption and improved the usability of the build by combining
> multiple workflows into a single larger workflow.
>
> The showstopper a year ago was the lack of being able to re-run a single
> failed job in a larger workflow.
> GitHub has since then delivered this feature and no showstoppers are
> present.
>
> I have been posting updates to
> https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows"
> about the progress.
> I have rebased the changes from last year's PoC, and I'm finalizing and
> testing the changes in my fork under
> https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to
> apache/pulsar, when the refactoring is ready.
>
This is great news !
Looking forward to your patch
Enrico
> -Lari
>
> On 2021/03/16 01:10:52 Sijie Guo wrote:
> > > The prototype has demonstrated about 60% reduction in
> > resource consumption.
> >
> > It is hard to quantify. Merging them into one large workflow can result
> in
> > more failures. Re-running those failures can consume resources as well.
> >
> > > Isn't it urgent to resolve it?
> >
> > I think we are in a stage that gives us breathing room to fix flaky tests
> > and solve other problems, no?
> > I don't mean we stop the effort here. I mean we have other enhancements
> > that we can do to improve the situation.
> > Once we get into a position where the flakiness is reduced, we can merge
> > them into one workflow.
> >
> > - Sijie
> >
> >
> > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> >
> > > Thanks for the feedback Sijie.
> > >
> > > > We are using a lazy consensus approach. Typically if there is no
> > > objection,
> > > > please go ahead and not need to wait for approval.
> > > > If people raise concerns, please address the concerns.
> > >
> > > You and Ali have raised concerns about changing the existing GitHub
> Actions
> > > workflows in a way where multiple workflows would be combined to a
> single
> > > workflow. Before proceeding, there is a need to address the concerns.
> We
> > > might end up with a completely different type of solution of what has
> been
> > > proposed initially. :)
> > >
> > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > workflows
> > > > into one giant workflow.
> > >
> > > I agree that addressing flaky tests is favorable. The main reason for
> PIP
> > > "Changes to GitHub Actions based Pulsar CI" is to
> > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > > 2) Reduce lead times for Pull Request feedback
> > > We cannot ignore these problems. If we don't change anything, the
> problems
> > > won't get fixed. The prototype has demonstrated about 60% reduction in
> > > resource consumption. Measuring the lead times hasn't been done in the
> > > prototype, but since the reason for long lead times has been long build
> > > queues due to excessive resource consumption, it's likely that the lead
> > > times would be reduced.
> > >
> > > I know that switching to a single workflow isn't the only solution to
> the
> > > above problems. I had a discussion with Ali. He recommended reducing
> the
> > > modules in Pulsar repository (PIP-62), reducing the docker container
> size
> > > and improving the Pulsar Broker unit test harness so that tests using
> it
> > > would be less flaky and that it would be easier to fix the issues in
> > > failing test when there would be better information about what was the
> > > state problem that caused the test to fail.
> > >
> > > As mentioned in the earlier email about the optimizations in the
> Pulsar CI
> > > refactoring prototype, the main benefits come from reusing binary
> artifacts
> > > from previous build stages so that each job doesn't have to build
> > > everything from scratch. This becomes irrelevant when the build is very
> > > fast and there isn't a benefit of reusing artifacts.
> > > This means that it's possible to resolve the resource consumption
> problem
> > > of Pulsar PR builds in the way that Ali is recommending, without
> switching
> > > from multiple workflows to a single workflow that can reuse binary
> > > artifacts in the build.
> > >
> > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > more
> > > > on solving flaky tests.
> > > > Consider merging them into one workflow when the tests are in a
> better
> > > > situation.
> > >
> > > Makes sense for minimizing the risk of change, but we cannot just wait
> for
> > > things to fix themselves.
> > > How long will other Apache projects tolerate the resource consumption
> > > issues Pulsar is causing in the shared GitHub Actions Runner VM quota?
> For
> > > example,
> https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > .
> > > Isn't it urgent to resolve it?
> > >
> > > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar
> CI"
> > > based on the community feedback in the upcoming days. That might mean
> that
> > > the current solution is pivoted. The goal is to solve the problems of
> high
> > > resource consumption and long lead time for PR build in Pulsar CI.
> Please
> > > continue to provide feedback so that we get a revisited plan together
> that
> > > addresses these problems.Thank you!
> > >
> > > BR,
> > > -Lari
> > >
> > > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> GitHub
> > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > page https://github.com/apache/pulsar/wiki
> > > > <https://github.com/apache/pulsar/wiki> ?*
> > > >
> > > > I see what you referred to before now. This can be easily done. I
> (or any
> > > > other committer) can do it for you.
> > > >
> > > > There is no real blocker for you to continue work even there are
> concerns
> > > > or it is not listed in the PIP.
> > > > We are using a lazy consensus approach. Typically if there is no
> > > objection,
> > > > please go ahead and not need to wait for approval.
> > > > If people raise concerns, please address the concerns.
> > > >
> > > > > The reason why
> > > > re-runs happen currently is because of the high flakiness of tests.
> > > >
> > > > Yes. So I am in favor of addressing flaky tests than merging all
> > > workflows
> > > > into one giant workflow.
> > > > It is not about "No pain, no gain". The community has suffered a lot
> with
> > > > giant workflow before.
> > > > There were a lot of committers and contributors working hard to
> split one
> > > > giant workflow into multiple
> > > > current workflows. Unless there is really strong evidence that
> merging
> > > them
> > > > back to one will improve
> > > > the entire CI experience, I will still have concerns about one giant
> > > > workflow approach.
> > > >
> > > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > > more
> > > > on solving flaky tests.
> > > > Consider merging them into one workflow when the tests are in a
> better
> > > > situation.
> > > >
> > > > > This solution would also require disabling
> > > > required status checks
> > > >
> > > > I don't think it is a good idea to disable status checks. We can
> consider
> > > > running "dark mode" but it will just overload the action quota.
> > > >
> > > > Another alternative is to mirror the pull requests into another
> Github
> > > > account to test that and get more concrete statistics on the
> flakiness
> > > rate
> > > > of one workflow approach.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > > >
> > > > > Thanks for the feedback, Sijie.
> > > > >
> > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > there
> > > > > are
> > > > > > multiple workflows and each workflow is retryable.
> > > > > > However, I am not sure how much it will help if you run all
> workflows
> > > > in
> > > > > > one giant workflow. Or is it making things worse?
> > > > >
> > > > > We can reduce the need for re-running workflow runs. The reason why
> > > > > re-runs happen currently is because of high flakiness of tests.
> > > > > Addressing flakiness continues to be top-priority. Now that the
> Pulsar
> > > CI
> > > > > workflow prototype is finished, I'll be focusing more in the other
> > > draft
> > > > > PIP, "Changes to flaky test handling".
> > > > > We as a community should address the critical problem that the
> current
> > > > > retry solution has: it can mask bugs in production code and make
> the
> > > > build
> > > > > pass and allow changes to be merged that cause regressions.
> > > > > It's a false sense of security what the green builds after all the
> > > > retries
> > > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > > addressing this.
> > > > >
> > > > > If something doesn't work, it can be adapted and improved. Changes
> can
> > > be
> > > > > rolled back and revisited when things go worse. We also need a
> leap of
> > > > > faith.
> > > > > "No pain, no gain", like any change, it will be painful at first,
> but
> > > we
> > > > > will get over the bump.
> > > > >
> > > > > > Secondly, your test has been done in your folk where there are
> not a
> > > > lot
> > > > > of
> > > > > > concurrent pushes and pull requests. I am not sure how your
> approach
> > > > will
> > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > concurrent
> > > > > > pull requests in your account to prove your approach doesn't
> bring
> > > side
> > > > > > effects?
> > > > >
> > > > > One possibility to address this is to introduce the new workflow
> in a
> > > > mode
> > > > > where you need to opt-in to the new workflow in some way.
> > > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > > It might be possible to configure the existing workflow and this
> new
> > > > > workflow in a way where some condition (for example whitelisted
> github
> > > > user
> > > > > name or a certain keyword in the PR title/description) chooses
> either
> > > one
> > > > > for the Pull request. This solution would also require disabling
> > > > > required status checks ("Require status checks to pass before
> merging"
> > > > > feature in GitHub branch protection rules) since the names of the
> > > checks
> > > > > would be different.
> > > > >
> > > > > > Lastly, can we apply those optimizations to current workflows
> without
> > > > > > merging them into one giant workflow?
> > > > >
> > > > > This is what I have been doing. All individual optimizations have
> > > already
> > > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > > have
> > > > > already been merged.
> > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > > There's also 2 build related PRs from yesterday which haven't been
> > > merged
> > > > > yet:
> > > > > Fix Maven download issues (ported from the prototype to our
> existing
> > > > Pulsar
> > > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > > improve Maven module build order (required for more efficient
> builds
> > > that
> > > > > selectively build required artifacts):
> > > > > https://github.com/apache/pulsar/pull/9882
> > > > >
> > > > > There aren't many optimizations left that could be ported from the
> > > > > prototype to the existing workflow. There are a few, but the
> impact is
> > > > > minor.
> > > > > The reason for this is that the optimization with the greatest
> impact
> > > are
> > > > > the ones that build a binary artifacts (maven libs, docker images)
> once
> > > > and
> > > > > share it with the downstream jobs in the pipeline.
> > > > > Applying this type of solution has certain challenges when there
> are
> > > > > multiple separate workflow. Sharing binary artifacts to other
> workflows
> > > > > would require that the workflow to reuse the artifacts gets
> triggered
> > > by
> > > > > the workflow that produced the artifacts. This wouldn't be secure
> or
> > > > > practical for handling pull requests. Triggering a workflow
> explicitly
> > > > > would require a token from the main repository and using that for
> pull
> > > > > request builds would be a serious security vulnerability. (more
> details
> > > > > about the GitHub Actions security model in
> > > > >
> > > > >
> > > >
> > >
> https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > > )
> > > > >
> > > > > *Sijie, how far are we from getting the draft PIP "Changes to
> GitHub
> > > > > Actions based Pulsar CI" into an actual PIP that gets put on the
> wiki
> > > > > page https://github.com/apache/pulsar/wiki
> > > > > <https://github.com/apache/pulsar/wiki> ?*
> > > > >
> > > > > -Lari
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com>
> wrote:
> > > > >
> > > > > > This is good progress. However, my main concern is still merging
> all
> > > > > > workflows into one giant workflow.
> > > > > >
> > > > > > The "Fail fast" approach is great. That would be super helpful if
> > > there
> > > > > are
> > > > > > multiple workflows and each workflow is retryable.
> > > > > > However, I am not sure how much it will help if you run all
> workflows
> > > > in
> > > > > > one giant workflow. Or is it making things worse?
> > > > > >
> > > > > > Secondly, your test has been done in your folk where there are
> not a
> > > > lot
> > > > > of
> > > > > > concurrent pushes and pull requests. I am not sure how your
> approach
> > > > will
> > > > > > behave once it is merged into master. Can you simulate multiple
> > > > > concurrent
> > > > > > pull requests in your account to prove your approach doesn't
> bring
> > > side
> > > > > > effects?
> > > > > >
> > > > > > Lastly, can we apply those optimizations to current workflows
> without
> > > > > > merging them into one giant workflow?
> > > > > >
> > > > > > - Sijie
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > > wrote:
> > > > > >
> > > > > > > Thanks for the feedback Michael.
> > > > > > >
> > > > > > > > I left a question on the doc about how concurrent runs
> affect the
> > > > > > > > repository's 5 GB cache limit.
> > > > > > >
> > > > > > > This is a good question. There isn't a clear answer in the
> GitHub
> > > > > Actions
> > > > > > > Cache documentation.
> > > > > > >
> > > > > > > The documentation is
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > > .
> > > > > > > Based on this document and some testing, I have made these
> > > > conclusions:
> > > > > > > For GitHub Actions Cache, pull requests get executed in the
> context
> > > > of
> > > > > > the
> > > > > > > forked repository.
> > > > > > > The workflow triggered by a pull request event can only update
> it's
> > > > own
> > > > > > > cache. It has read-only access to upstream caches.
> > > > > > > If there's a cache miss, the entry will get written to the
> cache of
> > > > the
> > > > > > > forked repository. If the PR could to write to the upstream
> cache,
> > > it
> > > > > > would
> > > > > > > be a security issue since this would be vulnerable to cache
> > > poisoning
> > > > > > > attacks. Each repository has a 5GB quota for writes. The
> entries
> > > are
> > > > > kept
> > > > > > > up to 7 days.
> > > > > > > The performance is fairly good. Loading docker images from the
> > > > > repository
> > > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s.
> (the
> > > > > > > performance of the GHA cache is most likely higher since this
> is
> > > the
> > > > > > > throughput for docker load / docker save)
> > > > > > >
> > > > > > > If a single repository has a lot of concurrent jobs, it could
> start
> > > > > > > evicting caches.
> > > > > > > However that isn't likely to happen with the way Pulsar is
> > > developed
> > > > > > since
> > > > > > > pull requests are created from personal forks.
> > > > > > >
> > > > > > > > I also think it could be helpful to explicitly document, or
> > > > reference
> > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > assuming
> > > > > > > that
> > > > > > > > if an action fails, its parallel peer actions will run to
> > > > completion,
> > > > > > and
> > > > > > > > that the rest of the remaining stages will get canceled, but
> I
> > > > > haven't
> > > > > > > > worked with github actions before.
> > > > > > >
> > > > > > > For matrix jobs, "fail fast" is the default, which cancels all
> jobs
> > > > in
> > > > > > the
> > > > > > > matrix if one fails. Other parallel flows would run to
> completion
> > > by
> > > > > > > default.
> > > > > > > In the prototype, I have added a Github script step to each
> job to
> > > > > cancel
> > > > > > > the complete workflow when a failure occurs.
> > > > > > > Here's an example:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > > >
> > > > > > > The prototype follows a "fail fast" design. When a failure
> occurs,
> > > > fail
> > > > > > > fast and don't continue with other jobs.
> > > > > > > The benefit of this is that it reduces resource consumption.
> This
> > > > helps
> > > > > > > keep the build queue short.
> > > > > > > When the build queue is short, developers get quick feedback
> from
> > > CI.
> > > > > > >
> > > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > > *I'm hoping to start a separate document on low level details
> when
> > > > > there
> > > > > > is
> > > > > > > a high level acceptance of the proposed "Changes to GitHub
> Actions
> > > > > based
> > > > > > > Pulsar CI".*
> > > > > > > Together we can make this happen. We need decisions too. This
> > > > proposal
> > > > > > > cannot stay as a draft forever.
> > > > > > > I'm looking forward to hearing from the Pulsar community,
> Pulsar
> > > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > > >
> > > > > > > BR,
> > > > > > > -Lari
> > > > > > >
> > > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > > mikemarsh17@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > This will be a great improvement. I read through the PIP, and
> > > > > overall,
> > > > > > it
> > > > > > > > looks good to me.
> > > > > > > >
> > > > > > > > I left a question on the doc about how concurrent runs
> affect the
> > > > > > > > repository's 5 GB cache limit.
> > > > > > > >
> > > > > > > > I also think it could be helpful to explicitly document, or
> > > > reference
> > > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > > assuming
> > > > > > > that
> > > > > > > > if an action fails, its parallel peer actions will run to
> > > > completion,
> > > > > > and
> > > > > > > > that the rest of the remaining stages will get canceled, but
> I
> > > > > haven't
> > > > > > > > worked with github actions before.
> > > > > > > >
> > > > > > > > Thanks for all of the work you've put in so far.
> > > > > > > >
> > > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > This is great news. Thanks Hari , Mateo and pulsar
> community
> > > > > > > > >
> > > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > > lari.hotari@sagire.fi
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > >
> > > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI"
> has
> > > > gone
> > > > > > > > forward
> > > > > > > > > > based on your feedback. Here are some updates about the
> work.
> > > > > > > > > >
> > > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > There's a *detailed status update in the document about a
> > > > > prototype
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > > >
> > > > > > > > > > Thanks for all the suggestions and feedback by now. A
> lot of
> > > > > > > > improvements
> > > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > > technical
> > > > > > > > > > obstacles.
> > > > > > > > > > Special thanks go to Matteo for reducing the sizes of
> docker
> > > > > > images.
> > > > > > > A
> > > > > > > > > lot
> > > > > > > > > > of small improvements have been made to the Pulsar maven
> > > build
> > > > to
> > > > > > > > enable
> > > > > > > > > > the new refactored GitHub Actions workflow. Thank you
> for all
> > > > PR
> > > > > > > > reviews
> > > > > > > > > > and feedback.
> > > > > > > > > >
> > > > > > > > > > The main goal of the "Changes to GitHub Actions based
> Pulsar
> > > > CI"
> > > > > > work
> > > > > > > > has
> > > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > > build
> > > > > and
> > > > > > > to
> > > > > > > > > > speed up Pulsar development by improving the developer
> > > > > > productivity*
> > > > > > > > when
> > > > > > > > > > less time is wasted in waiting for Pulsar CI build
> feedback.
> > > > The
> > > > > > > > > prototype
> > > > > > > > > > demonstrates these improvements.
> > > > > > > > > >
> > > > > > > > > > As you can see from the email from Jan 28 below, *the
> > > resource
> > > > > > > > > consumption
> > > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > > observed
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > work began.
> > > > > > > > > > Now, with the prototype of the refactored Pulsar CI
> build,
> > > the
> > > > > > > resource
> > > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > > *This is about 60% reduction in resource consumption.*
> The
> > > > whole
> > > > > > > > pipeline
> > > > > > > > > > completes in 75-100 minutes.
> > > > > > > > > >
> > > > > > > > > > Here's a breakdown of the duration (resource
> consumption) of
> > > > each
> > > > > > > build
> > > > > > > > > job
> > > > > > > > > > in the refactored workflow:
> > > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > > 0:06:45
> > > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968
> 0:16:08
> > > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223
> 0:37:03
> > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > > 0:14:09
> > > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > > 0:29:55
> > > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > > 0:38:32
> > > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > > *Total resource consumption*
> > > > > > > > > > 7:08:57
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > > ).
> > > > > > > > > > However, this is not a showstopper since there are ways
> to
> > > > > address
> > > > > > > the
> > > > > > > > > > issues that cause flakiness.
> > > > > > > > > > There is a separate PIP for changing the way to handle
> flaky
> > > > > tests.
> > > > > > > You
> > > > > > > > > can
> > > > > > > > > > find the link to that in the "Changes to GitHub Actions
> based
> > > > > > Pulsar
> > > > > > > > CI"
> > > > > > > > > > document's header.
> > > > > > > > > >
> > > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > > >
> > > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > > . *It also contains more details of the prototype that
> has
> > > been
> > > > > > > > > > successfully completed.*
> > > > > > > > > >
> > > > > > > > > > 2) *Please share your feedback and suggest a way
> forward.*
> > > > > > > > > >
> > > > > > > > > > *Thank you for your help!*
> > > > > > > > > >
> > > > > > > > > > BR, Lari
> > > > > > > > > >
> > > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > > lari.hotari@sagire.fi
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > >
> > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > consuming
> > > > > the
> > > > > > > > > majority
> > > > > > > > > > > of the shared pool of resources allocated for
> > > > > github.com/apache
> > > > > > > > > > projects.
> > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > demand
> > > > > to
> > > > > > > > > improve
> > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > <
> > > > > >
> https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > >
> > > > > > > > > > asap.
> > > > > > > > > > >
> > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > time
> > > > > > that a
> > > > > > > > > > Runner
> > > > > > > > > > > is occupied. I observed the workflow runs for handling
> a
> > > > single
> > > > > > > Pull
> > > > > > > > > > > Request (in my personal fork) and these were the
> running
> > > > > > durations:
> > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > >
> > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > Actions
> > > > > > > > Runners
> > > > > > > > > > is
> > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> request to
> > > > > > > > > > apache/pulsar.*
> > > > > > > > > > >
> > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> is
> > > very
> > > > > > high,
> > > > > > > > > this
> > > > > > > > > > > leads to the build queue to grow and take a long time
> to
> > > > > process.
> > > > > > > > > > >
> > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> for
> > > the
> > > > > > last
> > > > > > > 3
> > > > > > > > > > > months. During this period I worked on a few
> experiments.
> > > The
> > > > > > > > learnings
> > > > > > > > > > > from the past experiments are documented at a high
> level in
> > > > the
> > > > > > > > > following
> > > > > > > > > > > draft PIP document.
> > > > > > > > > > >
> > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> CI"
> > > > > > document
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > > Google doc:*
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > >
> > > > > > > > > > > *Please participate* so that we get the plan adjusted
> based
> > > > on
> > > > > > the
> > > > > > > > > > > feedback asap. If there's already a similar effort
> > > ongoing, I
> > > > > > hope
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > join efforts.
> > > > > > > > > > >
> > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > >
> > > > > > > > > > > BR, Lari
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <lh...@apache.org>.
I have resumed work to improve our GitHub Actions based Pulsar CI.
Last year, I worked on a proof-of-concept which significantly reduced the resource consumption and improved the usability of the build by combining multiple workflows into a single larger workflow.
The showstopper a year ago was the lack of being able to re-run a single failed job in a larger workflow.
GitHub has since then delivered this feature and no showstoppers are present.
I have been posting updates to https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows" about the progress.
I have rebased the changes from last year's PoC, and I'm finalizing and testing the changes in my fork under https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to apache/pulsar, when the refactoring is ready.
-Lari
On 2021/03/16 01:10:52 Sijie Guo wrote:
> > The prototype has demonstrated about 60% reduction in
> resource consumption.
>
> It is hard to quantify. Merging them into one large workflow can result in
> more failures. Re-running those failures can consume resources as well.
>
> > Isn't it urgent to resolve it?
>
> I think we are in a stage that gives us breathing room to fix flaky tests
> and solve other problems, no?
> I don't mean we stop the effort here. I mean we have other enhancements
> that we can do to improve the situation.
> Once we get into a position where the flakiness is reduced, we can merge
> them into one workflow.
>
> - Sijie
>
>
> On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
>
> > Thanks for the feedback Sijie.
> >
> > > We are using a lazy consensus approach. Typically if there is no
> > objection,
> > > please go ahead and not need to wait for approval.
> > > If people raise concerns, please address the concerns.
> >
> > You and Ali have raised concerns about changing the existing GitHub Actions
> > workflows in a way where multiple workflows would be combined to a single
> > workflow. Before proceeding, there is a need to address the concerns. We
> > might end up with a completely different type of solution of what has been
> > proposed initially. :)
> >
> > > Yes. So I am in favor of addressing flaky tests than merging all
> > workflows
> > > into one giant workflow.
> >
> > I agree that addressing flaky tests is favorable. The main reason for PIP
> > "Changes to GitHub Actions based Pulsar CI" is to
> > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> > 2) Reduce lead times for Pull Request feedback
> > We cannot ignore these problems. If we don't change anything, the problems
> > won't get fixed. The prototype has demonstrated about 60% reduction in
> > resource consumption. Measuring the lead times hasn't been done in the
> > prototype, but since the reason for long lead times has been long build
> > queues due to excessive resource consumption, it's likely that the lead
> > times would be reduced.
> >
> > I know that switching to a single workflow isn't the only solution to the
> > above problems. I had a discussion with Ali. He recommended reducing the
> > modules in Pulsar repository (PIP-62), reducing the docker container size
> > and improving the Pulsar Broker unit test harness so that tests using it
> > would be less flaky and that it would be easier to fix the issues in
> > failing test when there would be better information about what was the
> > state problem that caused the test to fail.
> >
> > As mentioned in the earlier email about the optimizations in the Pulsar CI
> > refactoring prototype, the main benefits come from reusing binary artifacts
> > from previous build stages so that each job doesn't have to build
> > everything from scratch. This becomes irrelevant when the build is very
> > fast and there isn't a benefit of reusing artifacts.
> > This means that it's possible to resolve the resource consumption problem
> > of Pulsar PR builds in the way that Ali is recommending, without switching
> > from multiple workflows to a single workflow that can reuse binary
> > artifacts in the build.
> >
> > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > more
> > > on solving flaky tests.
> > > Consider merging them into one workflow when the tests are in a better
> > > situation.
> >
> > Makes sense for minimizing the risk of change, but we cannot just wait for
> > things to fix themselves.
> > How long will other Apache projects tolerate the resource consumption
> > issues Pulsar is causing in the shared GitHub Actions Runner VM quota? For
> > example, https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > .
> > Isn't it urgent to resolve it?
> >
> > I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar CI"
> > based on the community feedback in the upcoming days. That might mean that
> > the current solution is pivoted. The goal is to solve the problems of high
> > resource consumption and long lead time for PR build in Pulsar CI. Please
> > continue to provide feedback so that we get a revisited plan together that
> > addresses these problems.Thank you!
> >
> > BR,
> > -Lari
> >
> > On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > > > *Sijie, how far are we from getting the draft PIP "Changes to GitHub
> > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > page https://github.com/apache/pulsar/wiki
> > > <https://github.com/apache/pulsar/wiki> ?*
> > >
> > > I see what you referred to before now. This can be easily done. I (or any
> > > other committer) can do it for you.
> > >
> > > There is no real blocker for you to continue work even there are concerns
> > > or it is not listed in the PIP.
> > > We are using a lazy consensus approach. Typically if there is no
> > objection,
> > > please go ahead and not need to wait for approval.
> > > If people raise concerns, please address the concerns.
> > >
> > > > The reason why
> > > re-runs happen currently is because of the high flakiness of tests.
> > >
> > > Yes. So I am in favor of addressing flaky tests than merging all
> > workflows
> > > into one giant workflow.
> > > It is not about "No pain, no gain". The community has suffered a lot with
> > > giant workflow before.
> > > There were a lot of committers and contributors working hard to split one
> > > giant workflow into multiple
> > > current workflows. Unless there is really strong evidence that merging
> > them
> > > back to one will improve
> > > the entire CI experience, I will still have concerns about one giant
> > > workflow approach.
> > >
> > > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> > more
> > > on solving flaky tests.
> > > Consider merging them into one workflow when the tests are in a better
> > > situation.
> > >
> > > > This solution would also require disabling
> > > required status checks
> > >
> > > I don't think it is a good idea to disable status checks. We can consider
> > > running "dark mode" but it will just overload the action quota.
> > >
> > > Another alternative is to mirror the pull requests into another Github
> > > account to test that and get more concrete statistics on the flakiness
> > rate
> > > of one workflow approach.
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> > >
> > > > Thanks for the feedback, Sijie.
> > > >
> > > > > The "Fail fast" approach is great. That would be super helpful if
> > there
> > > > are
> > > > > multiple workflows and each workflow is retryable.
> > > > > However, I am not sure how much it will help if you run all workflows
> > > in
> > > > > one giant workflow. Or is it making things worse?
> > > >
> > > > We can reduce the need for re-running workflow runs. The reason why
> > > > re-runs happen currently is because of high flakiness of tests.
> > > > Addressing flakiness continues to be top-priority. Now that the Pulsar
> > CI
> > > > workflow prototype is finished, I'll be focusing more in the other
> > draft
> > > > PIP, "Changes to flaky test handling".
> > > > We as a community should address the critical problem that the current
> > > > retry solution has: it can mask bugs in production code and make the
> > > build
> > > > pass and allow changes to be merged that cause regressions.
> > > > It's a false sense of security what the green builds after all the
> > > retries
> > > > bring us. Bringing Pulsar to the next level in stability requires
> > > > addressing this.
> > > >
> > > > If something doesn't work, it can be adapted and improved. Changes can
> > be
> > > > rolled back and revisited when things go worse. We also need a leap of
> > > > faith.
> > > > "No pain, no gain", like any change, it will be painful at first, but
> > we
> > > > will get over the bump.
> > > >
> > > > > Secondly, your test has been done in your folk where there are not a
> > > lot
> > > > of
> > > > > concurrent pushes and pull requests. I am not sure how your approach
> > > will
> > > > > behave once it is merged into master. Can you simulate multiple
> > > > concurrent
> > > > > pull requests in your account to prove your approach doesn't bring
> > side
> > > > > effects?
> > > >
> > > > One possibility to address this is to introduce the new workflow in a
> > > mode
> > > > where you need to opt-in to the new workflow in some way.
> > > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > > It might be possible to configure the existing workflow and this new
> > > > workflow in a way where some condition (for example whitelisted github
> > > user
> > > > name or a certain keyword in the PR title/description) chooses either
> > one
> > > > for the Pull request. This solution would also require disabling
> > > > required status checks ("Require status checks to pass before merging"
> > > > feature in GitHub branch protection rules) since the names of the
> > checks
> > > > would be different.
> > > >
> > > > > Lastly, can we apply those optimizations to current workflows without
> > > > > merging them into one giant workflow?
> > > >
> > > > This is what I have been doing. All individual optimizations have
> > already
> > > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> > have
> > > > already been merged.
> > > >
> > > >
> > >
> > https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > > There's also 2 build related PRs from yesterday which haven't been
> > merged
> > > > yet:
> > > > Fix Maven download issues (ported from the prototype to our existing
> > > Pulsar
> > > > CI): https://github.com/apache/pulsar/pull/9883
> > > > improve Maven module build order (required for more efficient builds
> > that
> > > > selectively build required artifacts):
> > > > https://github.com/apache/pulsar/pull/9882
> > > >
> > > > There aren't many optimizations left that could be ported from the
> > > > prototype to the existing workflow. There are a few, but the impact is
> > > > minor.
> > > > The reason for this is that the optimization with the greatest impact
> > are
> > > > the ones that build a binary artifacts (maven libs, docker images) once
> > > and
> > > > share it with the downstream jobs in the pipeline.
> > > > Applying this type of solution has certain challenges when there are
> > > > multiple separate workflow. Sharing binary artifacts to other workflows
> > > > would require that the workflow to reuse the artifacts gets triggered
> > by
> > > > the workflow that produced the artifacts. This wouldn't be secure or
> > > > practical for handling pull requests. Triggering a workflow explicitly
> > > > would require a token from the main repository and using that for pull
> > > > request builds would be a serious security vulnerability. (more details
> > > > about the GitHub Actions security model in
> > > >
> > > >
> > >
> > https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > > )
> > > >
> > > > *Sijie, how far are we from getting the draft PIP "Changes to GitHub
> > > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > > page https://github.com/apache/pulsar/wiki
> > > > <https://github.com/apache/pulsar/wiki> ?*
> > > >
> > > > -Lari
> > > >
> > > >
> > > >
> > > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com> wrote:
> > > >
> > > > > This is good progress. However, my main concern is still merging all
> > > > > workflows into one giant workflow.
> > > > >
> > > > > The "Fail fast" approach is great. That would be super helpful if
> > there
> > > > are
> > > > > multiple workflows and each workflow is retryable.
> > > > > However, I am not sure how much it will help if you run all workflows
> > > in
> > > > > one giant workflow. Or is it making things worse?
> > > > >
> > > > > Secondly, your test has been done in your folk where there are not a
> > > lot
> > > > of
> > > > > concurrent pushes and pull requests. I am not sure how your approach
> > > will
> > > > > behave once it is merged into master. Can you simulate multiple
> > > > concurrent
> > > > > pull requests in your account to prove your approach doesn't bring
> > side
> > > > > effects?
> > > > >
> > > > > Lastly, can we apply those optimizations to current workflows without
> > > > > merging them into one giant workflow?
> > > > >
> > > > > - Sijie
> > > > >
> > > > >
> > > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> > wrote:
> > > > >
> > > > > > Thanks for the feedback Michael.
> > > > > >
> > > > > > > I left a question on the doc about how concurrent runs affect the
> > > > > > > repository's 5 GB cache limit.
> > > > > >
> > > > > > This is a good question. There isn't a clear answer in the GitHub
> > > > Actions
> > > > > > Cache documentation.
> > > > > >
> > > > > > The documentation is
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > > .
> > > > > > Based on this document and some testing, I have made these
> > > conclusions:
> > > > > > For GitHub Actions Cache, pull requests get executed in the context
> > > of
> > > > > the
> > > > > > forked repository.
> > > > > > The workflow triggered by a pull request event can only update it's
> > > own
> > > > > > cache. It has read-only access to upstream caches.
> > > > > > If there's a cache miss, the entry will get written to the cache of
> > > the
> > > > > > forked repository. If the PR could to write to the upstream cache,
> > it
> > > > > would
> > > > > > be a security issue since this would be vulnerable to cache
> > poisoning
> > > > > > attacks. Each repository has a 5GB quota for writes. The entries
> > are
> > > > kept
> > > > > > up to 7 days.
> > > > > > The performance is fairly good. Loading docker images from the
> > > > repository
> > > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s. (the
> > > > > > performance of the GHA cache is most likely higher since this is
> > the
> > > > > > throughput for docker load / docker save)
> > > > > >
> > > > > > If a single repository has a lot of concurrent jobs, it could start
> > > > > > evicting caches.
> > > > > > However that isn't likely to happen with the way Pulsar is
> > developed
> > > > > since
> > > > > > pull requests are created from personal forks.
> > > > > >
> > > > > > > I also think it could be helpful to explicitly document, or
> > > reference
> > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > assuming
> > > > > > that
> > > > > > > if an action fails, its parallel peer actions will run to
> > > completion,
> > > > > and
> > > > > > > that the rest of the remaining stages will get canceled, but I
> > > > haven't
> > > > > > > worked with github actions before.
> > > > > >
> > > > > > For matrix jobs, "fail fast" is the default, which cancels all jobs
> > > in
> > > > > the
> > > > > > matrix if one fails. Other parallel flows would run to completion
> > by
> > > > > > default.
> > > > > > In the prototype, I have added a Github script step to each job to
> > > > cancel
> > > > > > the complete workflow when a failure occurs.
> > > > > > Here's an example:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > > >
> > > > > > The prototype follows a "fail fast" design. When a failure occurs,
> > > fail
> > > > > > fast and don't continue with other jobs.
> > > > > > The benefit of this is that it reduces resource consumption. This
> > > helps
> > > > > > keep the build queue short.
> > > > > > When the build queue is short, developers get quick feedback from
> > CI.
> > > > > >
> > > > > > Documenting all details in the PIP document isn't practical.
> > > > > > *I'm hoping to start a separate document on low level details when
> > > > there
> > > > > is
> > > > > > a high level acceptance of the proposed "Changes to GitHub Actions
> > > > based
> > > > > > Pulsar CI".*
> > > > > > Together we can make this happen. We need decisions too. This
> > > proposal
> > > > > > cannot stay as a draft forever.
> > > > > > I'm looking forward to hearing from the Pulsar community, Pulsar
> > > > > > committer and Pulsar PMC members how to take this forward.
> > > > > >
> > > > > > BR,
> > > > > > -Lari
> > > > > >
> > > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > > mikemarsh17@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > This will be a great improvement. I read through the PIP, and
> > > > overall,
> > > > > it
> > > > > > > looks good to me.
> > > > > > >
> > > > > > > I left a question on the doc about how concurrent runs affect the
> > > > > > > repository's 5 GB cache limit.
> > > > > > >
> > > > > > > I also think it could be helpful to explicitly document, or
> > > reference
> > > > > > > github documentation, on how failure will affect the DAG. I'm
> > > > assuming
> > > > > > that
> > > > > > > if an action fails, its parallel peer actions will run to
> > > completion,
> > > > > and
> > > > > > > that the rest of the remaining stages will get canceled, but I
> > > > haven't
> > > > > > > worked with github actions before.
> > > > > > >
> > > > > > > Thanks for all of the work you've put in so far.
> > > > > > >
> > > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > This is great news. Thanks Hari , Mateo and pulsar community
> > > > > > > >
> > > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> > lari.hotari@sagire.fi
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Dear Pulsar community members,
> > > > > > > > >
> > > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI" has
> > > gone
> > > > > > > forward
> > > > > > > > > based on your feedback. Here are some updates about the work.
> > > > > > > > >
> > > > > > > > > The draft PIP proposal document is here:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > There's a *detailed status update in the document about a
> > > > prototype
> > > > > > for
> > > > > > > > the
> > > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > > >
> > > > > > > > > Thanks for all the suggestions and feedback by now. A lot of
> > > > > > > improvements
> > > > > > > > > have been made by the Pulsar contributors to overcome the
> > > > technical
> > > > > > > > > obstacles.
> > > > > > > > > Special thanks go to Matteo for reducing the sizes of docker
> > > > > images.
> > > > > > A
> > > > > > > > lot
> > > > > > > > > of small improvements have been made to the Pulsar maven
> > build
> > > to
> > > > > > > enable
> > > > > > > > > the new refactored GitHub Actions workflow. Thank you for all
> > > PR
> > > > > > > reviews
> > > > > > > > > and feedback.
> > > > > > > > >
> > > > > > > > > The main goal of the "Changes to GitHub Actions based Pulsar
> > > CI"
> > > > > work
> > > > > > > has
> > > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> > build
> > > > and
> > > > > > to
> > > > > > > > > speed up Pulsar development by improving the developer
> > > > > productivity*
> > > > > > > when
> > > > > > > > > less time is wasted in waiting for Pulsar CI build feedback.
> > > The
> > > > > > > > prototype
> > > > > > > > > demonstrates these improvements.
> > > > > > > > >
> > > > > > > > > As you can see from the email from Jan 28 below, *the
> > resource
> > > > > > > > consumption
> > > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > > observed
> > > > > > when
> > > > > > > > the
> > > > > > > > > work began.
> > > > > > > > > Now, with the prototype of the refactored Pulsar CI build,
> > the
> > > > > > resource
> > > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > > *This is about 60% reduction in resource consumption.* The
> > > whole
> > > > > > > pipeline
> > > > > > > > > completes in 75-100 minutes.
> > > > > > > > >
> > > > > > > > > Here's a breakdown of the duration (resource consumption) of
> > > each
> > > > > > build
> > > > > > > > job
> > > > > > > > > in the refactored workflow:
> > > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> > 0:06:45
> > > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > > 0:14:09
> > > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> > 0:29:55
> > > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > > 0:38:32
> > > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > > *Total resource consumption*
> > > > > > > > > 7:08:57
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > > ).
> > > > > > > > > However, this is not a showstopper since there are ways to
> > > > address
> > > > > > the
> > > > > > > > > issues that cause flakiness.
> > > > > > > > > There is a separate PIP for changing the way to handle flaky
> > > > tests.
> > > > > > You
> > > > > > > > can
> > > > > > > > > find the link to that in the "Changes to GitHub Actions based
> > > > > Pulsar
> > > > > > > CI"
> > > > > > > > > document's header.
> > > > > > > > >
> > > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > > >
> > > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > > . *It also contains more details of the prototype that has
> > been
> > > > > > > > > successfully completed.*
> > > > > > > > >
> > > > > > > > > 2) *Please share your feedback and suggest a way forward.*
> > > > > > > > >
> > > > > > > > > *Thank you for your help!*
> > > > > > > > >
> > > > > > > > > BR, Lari
> > > > > > > > >
> > > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > > lari.hotari@sagire.fi
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > >
> > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > consuming
> > > > the
> > > > > > > > majority
> > > > > > > > > > of the shared pool of resources allocated for
> > > > github.com/apache
> > > > > > > > > projects.
> > > > > > > > > > Other Apache projects have been impacted and there is a
> > > demand
> > > > to
> > > > > > > > improve
> > > > > > > > > > the Pulsar CI
> > > > > > > > > > <
> > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > >
> > > > > > > > > asap.
> > > > > > > > > >
> > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > time
> > > > > that a
> > > > > > > > > Runner
> > > > > > > > > > is occupied. I observed the workflow runs for handling a
> > > single
> > > > > > Pull
> > > > > > > > > > Request (in my personal fork) and these were the running
> > > > > durations:
> > > > > > > > > > Workflow name Duration
> > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > >
> > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > Actions
> > > > > > > Runners
> > > > > > > > > is
> > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > > > > > > apache/pulsar.*
> > > > > > > > > >
> > > > > > > > > > Since GitHub Actions Runner resource pool utilization is
> > very
> > > > > high,
> > > > > > > > this
> > > > > > > > > > leads to the build queue to grow and take a long time to
> > > > process.
> > > > > > > > > >
> > > > > > > > > > I have been looking for ways to improve the Pulsar CI for
> > the
> > > > > last
> > > > > > 3
> > > > > > > > > > months. During this period I worked on a few experiments.
> > The
> > > > > > > learnings
> > > > > > > > > > from the past experiments are documented at a high level in
> > > the
> > > > > > > > following
> > > > > > > > > > draft PIP document.
> > > > > > > > > >
> > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> > > > > document
> > > > > > > is
> > > > > > > > a
> > > > > > > > > > Google doc:*
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > >
> > > > > > > > > > *Please participate* so that we get the plan adjusted based
> > > on
> > > > > the
> > > > > > > > > > feedback asap. If there's already a similar effort
> > ongoing, I
> > > > > hope
> > > > > > we
> > > > > > > > can
> > > > > > > > > > join efforts.
> > > > > > > > > >
> > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > >
> > > > > > > > > > BR, Lari
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
On Tue, Mar 16, 2021 at 3:11 AM Sijie Guo <gu...@gmail.com> wrote:
> > The prototype has demonstrated about 60% reduction in
> resource consumption.
>
> It is hard to quantify. Merging them into one large workflow can result in
> more failures. Re-running those failures can consume resources as well.
>
Yes, you are right.
>
> > Isn't it urgent to resolve it?
>
> I think we are in a stage that gives us breathing room to fix flaky tests
> and solve other problems, no?
>
I don't have access to the ASF infra-users mailing list where the resource
consumption problems have been discussed. I guess the problem isn't so bad
at the moment since it's not coming to us.
Yes, it makes sense to focus on the flaky test problem if the resource
consumption isn't a pressing problem.
> I don't mean we stop the effort here. I mean we have other enhancements
> that we can do to improve the situation.
> Once we get into a position where the flakiness is reduced, we can merge
> them into one workflow.
>
+1
Getting tests to pass with a lot of retries comes with a tradeoff. One of
the critical issues it causes is that real production issues might pass
tests and get masked as test flakiness. This causes regressions. The
tolerance of test flakiness results in more flaky tests being added to the
code base. Unless we make changes to "flaky test handling", we won't be
able to change the course.
Makes sense?
BR, Lari
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Sijie Guo <gu...@gmail.com>.
> The prototype has demonstrated about 60% reduction in
resource consumption.
It is hard to quantify. Merging them into one large workflow can result in
more failures. Re-running those failures can consume resources as well.
> Isn't it urgent to resolve it?
I think we are in a stage that gives us breathing room to fix flaky tests
and solve other problems, no?
I don't mean we stop the effort here. I mean we have other enhancements
that we can do to improve the situation.
Once we get into a position where the flakiness is reduced, we can merge
them into one workflow.
- Sijie
On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari <La...@hotari.net> wrote:
> Thanks for the feedback Sijie.
>
> > We are using a lazy consensus approach. Typically if there is no
> objection,
> > please go ahead and not need to wait for approval.
> > If people raise concerns, please address the concerns.
>
> You and Ali have raised concerns about changing the existing GitHub Actions
> workflows in a way where multiple workflows would be combined to a single
> workflow. Before proceeding, there is a need to address the concerns. We
> might end up with a completely different type of solution of what has been
> proposed initially. :)
>
> > Yes. So I am in favor of addressing flaky tests than merging all
> workflows
> > into one giant workflow.
>
> I agree that addressing flaky tests is favorable. The main reason for PIP
> "Changes to GitHub Actions based Pulsar CI" is to
> 1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
> 2) Reduce lead times for Pull Request feedback
> We cannot ignore these problems. If we don't change anything, the problems
> won't get fixed. The prototype has demonstrated about 60% reduction in
> resource consumption. Measuring the lead times hasn't been done in the
> prototype, but since the reason for long lead times has been long build
> queues due to excessive resource consumption, it's likely that the lead
> times would be reduced.
>
> I know that switching to a single workflow isn't the only solution to the
> above problems. I had a discussion with Ali. He recommended reducing the
> modules in Pulsar repository (PIP-62), reducing the docker container size
> and improving the Pulsar Broker unit test harness so that tests using it
> would be less flaky and that it would be easier to fix the issues in
> failing test when there would be better information about what was the
> state problem that caused the test to fail.
>
> As mentioned in the earlier email about the optimizations in the Pulsar CI
> refactoring prototype, the main benefits come from reusing binary artifacts
> from previous build stages so that each job doesn't have to build
> everything from scratch. This becomes irrelevant when the build is very
> fast and there isn't a benefit of reusing artifacts.
> This means that it's possible to resolve the resource consumption problem
> of Pulsar PR builds in the way that Ali is recommending, without switching
> from multiple workflows to a single workflow that can reuse binary
> artifacts in the build.
>
> > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> more
> > on solving flaky tests.
> > Consider merging them into one workflow when the tests are in a better
> > situation.
>
> Makes sense for minimizing the risk of change, but we cannot just wait for
> things to fix themselves.
> How long will other Apache projects tolerate the resource consumption
> issues Pulsar is causing in the shared GitHub Actions Runner VM quota? For
> example, https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> .
> Isn't it urgent to resolve it?
>
> I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar CI"
> based on the community feedback in the upcoming days. That might mean that
> the current solution is pivoted. The goal is to solve the problems of high
> resource consumption and long lead time for PR build in Pulsar CI. Please
> continue to provide feedback so that we get a revisited plan together that
> addresses these problems.Thank you!
>
> BR,
> -Lari
>
> On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
>
> > > *Sijie, how far are we from getting the draft PIP "Changes to GitHub
> > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > page https://github.com/apache/pulsar/wiki
> > <https://github.com/apache/pulsar/wiki> ?*
> >
> > I see what you referred to before now. This can be easily done. I (or any
> > other committer) can do it for you.
> >
> > There is no real blocker for you to continue work even there are concerns
> > or it is not listed in the PIP.
> > We are using a lazy consensus approach. Typically if there is no
> objection,
> > please go ahead and not need to wait for approval.
> > If people raise concerns, please address the concerns.
> >
> > > The reason why
> > re-runs happen currently is because of the high flakiness of tests.
> >
> > Yes. So I am in favor of addressing flaky tests than merging all
> workflows
> > into one giant workflow.
> > It is not about "No pain, no gain". The community has suffered a lot with
> > giant workflow before.
> > There were a lot of committers and contributors working hard to split one
> > giant workflow into multiple
> > current workflows. Unless there is really strong evidence that merging
> them
> > back to one will improve
> > the entire CI experience, I will still have concerns about one giant
> > workflow approach.
> >
> > Hence I am +1 to "changes to flaky test handing" and suggest focusing
> more
> > on solving flaky tests.
> > Consider merging them into one workflow when the tests are in a better
> > situation.
> >
> > > This solution would also require disabling
> > required status checks
> >
> > I don't think it is a good idea to disable status checks. We can consider
> > running "dark mode" but it will just overload the action quota.
> >
> > Another alternative is to mirror the pull requests into another Github
> > account to test that and get more concrete statistics on the flakiness
> rate
> > of one workflow approach.
> >
> >
> >
> >
> >
> >
> > On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> >
> > > Thanks for the feedback, Sijie.
> > >
> > > > The "Fail fast" approach is great. That would be super helpful if
> there
> > > are
> > > > multiple workflows and each workflow is retryable.
> > > > However, I am not sure how much it will help if you run all workflows
> > in
> > > > one giant workflow. Or is it making things worse?
> > >
> > > We can reduce the need for re-running workflow runs. The reason why
> > > re-runs happen currently is because of high flakiness of tests.
> > > Addressing flakiness continues to be top-priority. Now that the Pulsar
> CI
> > > workflow prototype is finished, I'll be focusing more in the other
> draft
> > > PIP, "Changes to flaky test handling".
> > > We as a community should address the critical problem that the current
> > > retry solution has: it can mask bugs in production code and make the
> > build
> > > pass and allow changes to be merged that cause regressions.
> > > It's a false sense of security what the green builds after all the
> > retries
> > > bring us. Bringing Pulsar to the next level in stability requires
> > > addressing this.
> > >
> > > If something doesn't work, it can be adapted and improved. Changes can
> be
> > > rolled back and revisited when things go worse. We also need a leap of
> > > faith.
> > > "No pain, no gain", like any change, it will be painful at first, but
> we
> > > will get over the bump.
> > >
> > > > Secondly, your test has been done in your folk where there are not a
> > lot
> > > of
> > > > concurrent pushes and pull requests. I am not sure how your approach
> > will
> > > > behave once it is merged into master. Can you simulate multiple
> > > concurrent
> > > > pull requests in your account to prove your approach doesn't bring
> side
> > > > effects?
> > >
> > > One possibility to address this is to introduce the new workflow in a
> > mode
> > > where you need to opt-in to the new workflow in some way.
> > > This was an idea brought up by my colleagues Enrico and Andrey.
> > > It might be possible to configure the existing workflow and this new
> > > workflow in a way where some condition (for example whitelisted github
> > user
> > > name or a certain keyword in the PR title/description) chooses either
> one
> > > for the Pull request. This solution would also require disabling
> > > required status checks ("Require status checks to pass before merging"
> > > feature in GitHub branch protection rules) since the names of the
> checks
> > > would be different.
> > >
> > > > Lastly, can we apply those optimizations to current workflows without
> > > > merging them into one giant workflow?
> > >
> > > This is what I have been doing. All individual optimizations have
> already
> > > been sent as PRs in the last months. I guess there's 20-30 PRs that
> have
> > > already been merged.
> > >
> > >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > > There's also 2 build related PRs from yesterday which haven't been
> merged
> > > yet:
> > > Fix Maven download issues (ported from the prototype to our existing
> > Pulsar
> > > CI): https://github.com/apache/pulsar/pull/9883
> > > improve Maven module build order (required for more efficient builds
> that
> > > selectively build required artifacts):
> > > https://github.com/apache/pulsar/pull/9882
> > >
> > > There aren't many optimizations left that could be ported from the
> > > prototype to the existing workflow. There are a few, but the impact is
> > > minor.
> > > The reason for this is that the optimization with the greatest impact
> are
> > > the ones that build a binary artifacts (maven libs, docker images) once
> > and
> > > share it with the downstream jobs in the pipeline.
> > > Applying this type of solution has certain challenges when there are
> > > multiple separate workflow. Sharing binary artifacts to other workflows
> > > would require that the workflow to reuse the artifacts gets triggered
> by
> > > the workflow that produced the artifacts. This wouldn't be secure or
> > > practical for handling pull requests. Triggering a workflow explicitly
> > > would require a token from the main repository and using that for pull
> > > request builds would be a serious security vulnerability. (more details
> > > about the GitHub Actions security model in
> > >
> > >
> >
> https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > > )
> > >
> > > *Sijie, how far are we from getting the draft PIP "Changes to GitHub
> > > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > > page https://github.com/apache/pulsar/wiki
> > > <https://github.com/apache/pulsar/wiki> ?*
> > >
> > > -Lari
> > >
> > >
> > >
> > > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > > This is good progress. However, my main concern is still merging all
> > > > workflows into one giant workflow.
> > > >
> > > > The "Fail fast" approach is great. That would be super helpful if
> there
> > > are
> > > > multiple workflows and each workflow is retryable.
> > > > However, I am not sure how much it will help if you run all workflows
> > in
> > > > one giant workflow. Or is it making things worse?
> > > >
> > > > Secondly, your test has been done in your folk where there are not a
> > lot
> > > of
> > > > concurrent pushes and pull requests. I am not sure how your approach
> > will
> > > > behave once it is merged into master. Can you simulate multiple
> > > concurrent
> > > > pull requests in your account to prove your approach doesn't bring
> side
> > > > effects?
> > > >
> > > > Lastly, can we apply those optimizations to current workflows without
> > > > merging them into one giant workflow?
> > > >
> > > > - Sijie
> > > >
> > > >
> > > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net>
> wrote:
> > > >
> > > > > Thanks for the feedback Michael.
> > > > >
> > > > > > I left a question on the doc about how concurrent runs affect the
> > > > > > repository's 5 GB cache limit.
> > > > >
> > > > > This is a good question. There isn't a clear answer in the GitHub
> > > Actions
> > > > > Cache documentation.
> > > > >
> > > > > The documentation is
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > > .
> > > > > Based on this document and some testing, I have made these
> > conclusions:
> > > > > For GitHub Actions Cache, pull requests get executed in the context
> > of
> > > > the
> > > > > forked repository.
> > > > > The workflow triggered by a pull request event can only update it's
> > own
> > > > > cache. It has read-only access to upstream caches.
> > > > > If there's a cache miss, the entry will get written to the cache of
> > the
> > > > > forked repository. If the PR could to write to the upstream cache,
> it
> > > > would
> > > > > be a security issue since this would be vulnerable to cache
> poisoning
> > > > > attacks. Each repository has a 5GB quota for writes. The entries
> are
> > > kept
> > > > > up to 7 days.
> > > > > The performance is fairly good. Loading docker images from the
> > > repository
> > > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s. (the
> > > > > performance of the GHA cache is most likely higher since this is
> the
> > > > > throughput for docker load / docker save)
> > > > >
> > > > > If a single repository has a lot of concurrent jobs, it could start
> > > > > evicting caches.
> > > > > However that isn't likely to happen with the way Pulsar is
> developed
> > > > since
> > > > > pull requests are created from personal forks.
> > > > >
> > > > > > I also think it could be helpful to explicitly document, or
> > reference
> > > > > > github documentation, on how failure will affect the DAG. I'm
> > > assuming
> > > > > that
> > > > > > if an action fails, its parallel peer actions will run to
> > completion,
> > > > and
> > > > > > that the rest of the remaining stages will get canceled, but I
> > > haven't
> > > > > > worked with github actions before.
> > > > >
> > > > > For matrix jobs, "fail fast" is the default, which cancels all jobs
> > in
> > > > the
> > > > > matrix if one fails. Other parallel flows would run to completion
> by
> > > > > default.
> > > > > In the prototype, I have added a Github script step to each job to
> > > cancel
> > > > > the complete workflow when a failure occurs.
> > > > > Here's an example:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > > >
> > > > > The prototype follows a "fail fast" design. When a failure occurs,
> > fail
> > > > > fast and don't continue with other jobs.
> > > > > The benefit of this is that it reduces resource consumption. This
> > helps
> > > > > keep the build queue short.
> > > > > When the build queue is short, developers get quick feedback from
> CI.
> > > > >
> > > > > Documenting all details in the PIP document isn't practical.
> > > > > *I'm hoping to start a separate document on low level details when
> > > there
> > > > is
> > > > > a high level acceptance of the proposed "Changes to GitHub Actions
> > > based
> > > > > Pulsar CI".*
> > > > > Together we can make this happen. We need decisions too. This
> > proposal
> > > > > cannot stay as a draft forever.
> > > > > I'm looking forward to hearing from the Pulsar community, Pulsar
> > > > > committer and Pulsar PMC members how to take this forward.
> > > > >
> > > > > BR,
> > > > > -Lari
> > > > >
> > > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > > mikemarsh17@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This will be a great improvement. I read through the PIP, and
> > > overall,
> > > > it
> > > > > > looks good to me.
> > > > > >
> > > > > > I left a question on the doc about how concurrent runs affect the
> > > > > > repository's 5 GB cache limit.
> > > > > >
> > > > > > I also think it could be helpful to explicitly document, or
> > reference
> > > > > > github documentation, on how failure will affect the DAG. I'm
> > > assuming
> > > > > that
> > > > > > if an action fails, its parallel peer actions will run to
> > completion,
> > > > and
> > > > > > that the rest of the remaining stages will get canceled, but I
> > > haven't
> > > > > > worked with github actions before.
> > > > > >
> > > > > > Thanks for all of the work you've put in so far.
> > > > > >
> > > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > This is great news. Thanks Hari , Mateo and pulsar community
> > > > > > >
> > > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <
> lari.hotari@sagire.fi
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Dear Pulsar community members,
> > > > > > > >
> > > > > > > > The work on "Changes to GitHub Actions based Pulsar CI" has
> > gone
> > > > > > forward
> > > > > > > > based on your feedback. Here are some updates about the work.
> > > > > > > >
> > > > > > > > The draft PIP proposal document is here:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > There's a *detailed status update in the document about a
> > > prototype
> > > > > for
> > > > > > > the
> > > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > > >
> > > > > > > > Thanks for all the suggestions and feedback by now. A lot of
> > > > > > improvements
> > > > > > > > have been made by the Pulsar contributors to overcome the
> > > technical
> > > > > > > > obstacles.
> > > > > > > > Special thanks go to Matteo for reducing the sizes of docker
> > > > images.
> > > > > A
> > > > > > > lot
> > > > > > > > of small improvements have been made to the Pulsar maven
> build
> > to
> > > > > > enable
> > > > > > > > the new refactored GitHub Actions workflow. Thank you for all
> > PR
> > > > > > reviews
> > > > > > > > and feedback.
> > > > > > > >
> > > > > > > > The main goal of the "Changes to GitHub Actions based Pulsar
> > CI"
> > > > work
> > > > > > has
> > > > > > > > been to *reduce the resource consumption of the Pulsar CI
> build
> > > and
> > > > > to
> > > > > > > > speed up Pulsar development by improving the developer
> > > > productivity*
> > > > > > when
> > > > > > > > less time is wasted in waiting for Pulsar CI build feedback.
> > The
> > > > > > > prototype
> > > > > > > > demonstrates these improvements.
> > > > > > > >
> > > > > > > > As you can see from the email from Jan 28 below, *the
> resource
> > > > > > > consumption
> > > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > > observed
> > > > > when
> > > > > > > the
> > > > > > > > work began.
> > > > > > > > Now, with the prototype of the refactored Pulsar CI build,
> the
> > > > > resource
> > > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > > *This is about 60% reduction in resource consumption.* The
> > whole
> > > > > > pipeline
> > > > > > > > completes in 75-100 minutes.
> > > > > > > >
> > > > > > > > Here's a breakdown of the duration (resource consumption) of
> > each
> > > > > build
> > > > > > > job
> > > > > > > > in the refactored workflow:
> > > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405
> 0:06:45
> > > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> > 0:14:09
> > > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795
> 0:29:55
> > > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> > 0:38:32
> > > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > > *Total resource consumption*
> > > > > > > > 7:08:57
> > > > > > > >
> > > > > > > >
> > > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > > ).
> > > > > > > > However, this is not a showstopper since there are ways to
> > > address
> > > > > the
> > > > > > > > issues that cause flakiness.
> > > > > > > > There is a separate PIP for changing the way to handle flaky
> > > tests.
> > > > > You
> > > > > > > can
> > > > > > > > find the link to that in the "Changes to GitHub Actions based
> > > > Pulsar
> > > > > > CI"
> > > > > > > > document's header.
> > > > > > > >
> > > > > > > > *Some requests for the Pulsar community:*
> > > > > > > >
> > > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > > . *It also contains more details of the prototype that has
> been
> > > > > > > > successfully completed.*
> > > > > > > >
> > > > > > > > 2) *Please share your feedback and suggest a way forward.*
> > > > > > > >
> > > > > > > > *Thank you for your help!*
> > > > > > > >
> > > > > > > > BR, Lari
> > > > > > > >
> > > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > > lari.hotari@sagire.fi
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Dear Pulsar community members,
> > > > > > > > >
> > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> consuming
> > > the
> > > > > > > majority
> > > > > > > > > of the shared pool of resources allocated for
> > > github.com/apache
> > > > > > > > projects.
> > > > > > > > > Other Apache projects have been impacted and there is a
> > demand
> > > to
> > > > > > > improve
> > > > > > > > > the Pulsar CI
> > > > > > > > > <
> > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > >
> > > > > > > > asap.
> > > > > > > > >
> > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> time
> > > > that a
> > > > > > > > Runner
> > > > > > > > > is occupied. I observed the workflow runs for handling a
> > single
> > > > > Pull
> > > > > > > > > Request (in my personal fork) and these were the running
> > > > durations:
> > > > > > > > > Workflow name Duration
> > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > TOTAL 19:36:50
> > > > > > > > >
> > > > > > > > > *In this case, the total resource consumption of GitHub
> > Actions
> > > > > > Runners
> > > > > > > > is
> > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > > > > > apache/pulsar.*
> > > > > > > > >
> > > > > > > > > Since GitHub Actions Runner resource pool utilization is
> very
> > > > high,
> > > > > > > this
> > > > > > > > > leads to the build queue to grow and take a long time to
> > > process.
> > > > > > > > >
> > > > > > > > > I have been looking for ways to improve the Pulsar CI for
> the
> > > > last
> > > > > 3
> > > > > > > > > months. During this period I worked on a few experiments.
> The
> > > > > > learnings
> > > > > > > > > from the past experiments are documented at a high level in
> > the
> > > > > > > following
> > > > > > > > > draft PIP document.
> > > > > > > > >
> > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> > > > document
> > > > > > is
> > > > > > > a
> > > > > > > > > Google doc:*
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > >
> > > > > > > > > *Please participate* so that we get the plan adjusted based
> > on
> > > > the
> > > > > > > > > feedback asap. If there's already a similar effort
> ongoing, I
> > > > hope
> > > > > we
> > > > > > > can
> > > > > > > > > join efforts.
> > > > > > > > >
> > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > >
> > > > > > > > > BR, Lari
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Thanks for the feedback Sijie.
> We are using a lazy consensus approach. Typically if there is no
objection,
> please go ahead and not need to wait for approval.
> If people raise concerns, please address the concerns.
You and Ali have raised concerns about changing the existing GitHub Actions
workflows in a way where multiple workflows would be combined to a single
workflow. Before proceeding, there is a need to address the concerns. We
might end up with a completely different type of solution of what has been
proposed initially. :)
> Yes. So I am in favor of addressing flaky tests than merging all workflows
> into one giant workflow.
I agree that addressing flaky tests is favorable. The main reason for PIP
"Changes to GitHub Actions based Pulsar CI" is to
1) Reduce GitHub Action Runner resource consumption of Pulsar PR builds
2) Reduce lead times for Pull Request feedback
We cannot ignore these problems. If we don't change anything, the problems
won't get fixed. The prototype has demonstrated about 60% reduction in
resource consumption. Measuring the lead times hasn't been done in the
prototype, but since the reason for long lead times has been long build
queues due to excessive resource consumption, it's likely that the lead
times would be reduced.
I know that switching to a single workflow isn't the only solution to the
above problems. I had a discussion with Ali. He recommended reducing the
modules in Pulsar repository (PIP-62), reducing the docker container size
and improving the Pulsar Broker unit test harness so that tests using it
would be less flaky and that it would be easier to fix the issues in
failing test when there would be better information about what was the
state problem that caused the test to fail.
As mentioned in the earlier email about the optimizations in the Pulsar CI
refactoring prototype, the main benefits come from reusing binary artifacts
from previous build stages so that each job doesn't have to build
everything from scratch. This becomes irrelevant when the build is very
fast and there isn't a benefit of reusing artifacts.
This means that it's possible to resolve the resource consumption problem
of Pulsar PR builds in the way that Ali is recommending, without switching
from multiple workflows to a single workflow that can reuse binary
artifacts in the build.
> Hence I am +1 to "changes to flaky test handing" and suggest focusing more
> on solving flaky tests.
> Consider merging them into one workflow when the tests are in a better
> situation.
Makes sense for minimizing the risk of change, but we cannot just wait for
things to fix themselves.
How long will other Apache projects tolerate the resource consumption
issues Pulsar is causing in the shared GitHub Actions Runner VM quota? For
example, https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
.
Isn't it urgent to resolve it?
I'll revisit the plan for PIP "Changes to GitHub Actions based Pulsar CI"
based on the community feedback in the upcoming days. That might mean that
the current solution is pivoted. The goal is to solve the problems of high
resource consumption and long lead time for PR build in Pulsar CI. Please
continue to provide feedback so that we get a revisited plan together that
addresses these problems.Thank you!
BR,
-Lari
On Fri, Mar 12, 2021 at 11:06 PM Sijie Guo <gu...@gmail.com> wrote:
> > *Sijie, how far are we from getting the draft PIP "Changes to GitHub
> Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> page https://github.com/apache/pulsar/wiki
> <https://github.com/apache/pulsar/wiki> ?*
>
> I see what you referred to before now. This can be easily done. I (or any
> other committer) can do it for you.
>
> There is no real blocker for you to continue work even there are concerns
> or it is not listed in the PIP.
> We are using a lazy consensus approach. Typically if there is no objection,
> please go ahead and not need to wait for approval.
> If people raise concerns, please address the concerns.
>
> > The reason why
> re-runs happen currently is because of the high flakiness of tests.
>
> Yes. So I am in favor of addressing flaky tests than merging all workflows
> into one giant workflow.
> It is not about "No pain, no gain". The community has suffered a lot with
> giant workflow before.
> There were a lot of committers and contributors working hard to split one
> giant workflow into multiple
> current workflows. Unless there is really strong evidence that merging them
> back to one will improve
> the entire CI experience, I will still have concerns about one giant
> workflow approach.
>
> Hence I am +1 to "changes to flaky test handing" and suggest focusing more
> on solving flaky tests.
> Consider merging them into one workflow when the tests are in a better
> situation.
>
> > This solution would also require disabling
> required status checks
>
> I don't think it is a good idea to disable status checks. We can consider
> running "dark mode" but it will just overload the action quota.
>
> Another alternative is to mirror the pull requests into another Github
> account to test that and get more concrete statistics on the flakiness rate
> of one workflow approach.
>
>
>
>
>
>
> On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
>
> > Thanks for the feedback, Sijie.
> >
> > > The "Fail fast" approach is great. That would be super helpful if there
> > are
> > > multiple workflows and each workflow is retryable.
> > > However, I am not sure how much it will help if you run all workflows
> in
> > > one giant workflow. Or is it making things worse?
> >
> > We can reduce the need for re-running workflow runs. The reason why
> > re-runs happen currently is because of high flakiness of tests.
> > Addressing flakiness continues to be top-priority. Now that the Pulsar CI
> > workflow prototype is finished, I'll be focusing more in the other draft
> > PIP, "Changes to flaky test handling".
> > We as a community should address the critical problem that the current
> > retry solution has: it can mask bugs in production code and make the
> build
> > pass and allow changes to be merged that cause regressions.
> > It's a false sense of security what the green builds after all the
> retries
> > bring us. Bringing Pulsar to the next level in stability requires
> > addressing this.
> >
> > If something doesn't work, it can be adapted and improved. Changes can be
> > rolled back and revisited when things go worse. We also need a leap of
> > faith.
> > "No pain, no gain", like any change, it will be painful at first, but we
> > will get over the bump.
> >
> > > Secondly, your test has been done in your folk where there are not a
> lot
> > of
> > > concurrent pushes and pull requests. I am not sure how your approach
> will
> > > behave once it is merged into master. Can you simulate multiple
> > concurrent
> > > pull requests in your account to prove your approach doesn't bring side
> > > effects?
> >
> > One possibility to address this is to introduce the new workflow in a
> mode
> > where you need to opt-in to the new workflow in some way.
> > This was an idea brought up by my colleagues Enrico and Andrey.
> > It might be possible to configure the existing workflow and this new
> > workflow in a way where some condition (for example whitelisted github
> user
> > name or a certain keyword in the PR title/description) chooses either one
> > for the Pull request. This solution would also require disabling
> > required status checks ("Require status checks to pass before merging"
> > feature in GitHub branch protection rules) since the names of the checks
> > would be different.
> >
> > > Lastly, can we apply those optimizations to current workflows without
> > > merging them into one giant workflow?
> >
> > This is what I have been doing. All individual optimizations have already
> > been sent as PRs in the last months. I guess there's 20-30 PRs that have
> > already been merged.
> >
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> > There's also 2 build related PRs from yesterday which haven't been merged
> > yet:
> > Fix Maven download issues (ported from the prototype to our existing
> Pulsar
> > CI): https://github.com/apache/pulsar/pull/9883
> > improve Maven module build order (required for more efficient builds that
> > selectively build required artifacts):
> > https://github.com/apache/pulsar/pull/9882
> >
> > There aren't many optimizations left that could be ported from the
> > prototype to the existing workflow. There are a few, but the impact is
> > minor.
> > The reason for this is that the optimization with the greatest impact are
> > the ones that build a binary artifacts (maven libs, docker images) once
> and
> > share it with the downstream jobs in the pipeline.
> > Applying this type of solution has certain challenges when there are
> > multiple separate workflow. Sharing binary artifacts to other workflows
> > would require that the workflow to reuse the artifacts gets triggered by
> > the workflow that produced the artifacts. This wouldn't be secure or
> > practical for handling pull requests. Triggering a workflow explicitly
> > would require a token from the main repository and using that for pull
> > request builds would be a serious security vulnerability. (more details
> > about the GitHub Actions security model in
> >
> >
> https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> > )
> >
> > *Sijie, how far are we from getting the draft PIP "Changes to GitHub
> > Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> > page https://github.com/apache/pulsar/wiki
> > <https://github.com/apache/pulsar/wiki> ?*
> >
> > -Lari
> >
> >
> >
> > On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com> wrote:
> >
> > > This is good progress. However, my main concern is still merging all
> > > workflows into one giant workflow.
> > >
> > > The "Fail fast" approach is great. That would be super helpful if there
> > are
> > > multiple workflows and each workflow is retryable.
> > > However, I am not sure how much it will help if you run all workflows
> in
> > > one giant workflow. Or is it making things worse?
> > >
> > > Secondly, your test has been done in your folk where there are not a
> lot
> > of
> > > concurrent pushes and pull requests. I am not sure how your approach
> will
> > > behave once it is merged into master. Can you simulate multiple
> > concurrent
> > > pull requests in your account to prove your approach doesn't bring side
> > > effects?
> > >
> > > Lastly, can we apply those optimizations to current workflows without
> > > merging them into one giant workflow?
> > >
> > > - Sijie
> > >
> > >
> > > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net> wrote:
> > >
> > > > Thanks for the feedback Michael.
> > > >
> > > > > I left a question on the doc about how concurrent runs affect the
> > > > > repository's 5 GB cache limit.
> > > >
> > > > This is a good question. There isn't a clear answer in the GitHub
> > Actions
> > > > Cache documentation.
> > > >
> > > > The documentation is
> > > >
> > > >
> > >
> >
> https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > > .
> > > > Based on this document and some testing, I have made these
> conclusions:
> > > > For GitHub Actions Cache, pull requests get executed in the context
> of
> > > the
> > > > forked repository.
> > > > The workflow triggered by a pull request event can only update it's
> own
> > > > cache. It has read-only access to upstream caches.
> > > > If there's a cache miss, the entry will get written to the cache of
> the
> > > > forked repository. If the PR could to write to the upstream cache, it
> > > would
> > > > be a security issue since this would be vulnerable to cache poisoning
> > > > attacks. Each repository has a 5GB quota for writes. The entries are
> > kept
> > > > up to 7 days.
> > > > The performance is fairly good. Loading docker images from the
> > repository
> > > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s. (the
> > > > performance of the GHA cache is most likely higher since this is the
> > > > throughput for docker load / docker save)
> > > >
> > > > If a single repository has a lot of concurrent jobs, it could start
> > > > evicting caches.
> > > > However that isn't likely to happen with the way Pulsar is developed
> > > since
> > > > pull requests are created from personal forks.
> > > >
> > > > > I also think it could be helpful to explicitly document, or
> reference
> > > > > github documentation, on how failure will affect the DAG. I'm
> > assuming
> > > > that
> > > > > if an action fails, its parallel peer actions will run to
> completion,
> > > and
> > > > > that the rest of the remaining stages will get canceled, but I
> > haven't
> > > > > worked with github actions before.
> > > >
> > > > For matrix jobs, "fail fast" is the default, which cancels all jobs
> in
> > > the
> > > > matrix if one fails. Other parallel flows would run to completion by
> > > > default.
> > > > In the prototype, I have added a Github script step to each job to
> > cancel
> > > > the complete workflow when a failure occurs.
> > > > Here's an example:
> > > >
> > > >
> > >
> >
> https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > > >
> > > > The prototype follows a "fail fast" design. When a failure occurs,
> fail
> > > > fast and don't continue with other jobs.
> > > > The benefit of this is that it reduces resource consumption. This
> helps
> > > > keep the build queue short.
> > > > When the build queue is short, developers get quick feedback from CI.
> > > >
> > > > Documenting all details in the PIP document isn't practical.
> > > > *I'm hoping to start a separate document on low level details when
> > there
> > > is
> > > > a high level acceptance of the proposed "Changes to GitHub Actions
> > based
> > > > Pulsar CI".*
> > > > Together we can make this happen. We need decisions too. This
> proposal
> > > > cannot stay as a draft forever.
> > > > I'm looking forward to hearing from the Pulsar community, Pulsar
> > > > committer and Pulsar PMC members how to take this forward.
> > > >
> > > > BR,
> > > > -Lari
> > > >
> > > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> > mikemarsh17@gmail.com>
> > > > wrote:
> > > >
> > > > > This will be a great improvement. I read through the PIP, and
> > overall,
> > > it
> > > > > looks good to me.
> > > > >
> > > > > I left a question on the doc about how concurrent runs affect the
> > > > > repository's 5 GB cache limit.
> > > > >
> > > > > I also think it could be helpful to explicitly document, or
> reference
> > > > > github documentation, on how failure will affect the DAG. I'm
> > assuming
> > > > that
> > > > > if an action fails, its parallel peer actions will run to
> completion,
> > > and
> > > > > that the rest of the remaining stages will get canceled, but I
> > haven't
> > > > > worked with github actions before.
> > > > >
> > > > > Thanks for all of the work you've put in so far.
> > > > >
> > > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com>
> wrote:
> > > > >
> > > > > > This is great news. Thanks Hari , Mateo and pulsar community
> > > > > >
> > > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <lari.hotari@sagire.fi
> >
> > > > wrote:
> > > > > >
> > > > > > > Dear Pulsar community members,
> > > > > > >
> > > > > > > The work on "Changes to GitHub Actions based Pulsar CI" has
> gone
> > > > > forward
> > > > > > > based on your feedback. Here are some updates about the work.
> > > > > > >
> > > > > > > The draft PIP proposal document is here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > There's a *detailed status update in the document about a
> > prototype
> > > > for
> > > > > > the
> > > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > > >
> > > > > > > Thanks for all the suggestions and feedback by now. A lot of
> > > > > improvements
> > > > > > > have been made by the Pulsar contributors to overcome the
> > technical
> > > > > > > obstacles.
> > > > > > > Special thanks go to Matteo for reducing the sizes of docker
> > > images.
> > > > A
> > > > > > lot
> > > > > > > of small improvements have been made to the Pulsar maven build
> to
> > > > > enable
> > > > > > > the new refactored GitHub Actions workflow. Thank you for all
> PR
> > > > > reviews
> > > > > > > and feedback.
> > > > > > >
> > > > > > > The main goal of the "Changes to GitHub Actions based Pulsar
> CI"
> > > work
> > > > > has
> > > > > > > been to *reduce the resource consumption of the Pulsar CI build
> > and
> > > > to
> > > > > > > speed up Pulsar development by improving the developer
> > > productivity*
> > > > > when
> > > > > > > less time is wasted in waiting for Pulsar CI build feedback.
> The
> > > > > > prototype
> > > > > > > demonstrates these improvements.
> > > > > > >
> > > > > > > As you can see from the email from Jan 28 below, *the resource
> > > > > > consumption
> > > > > > > was 19 hrs 36 minutes* for a single pull request that was
> > observed
> > > > when
> > > > > > the
> > > > > > > work began.
> > > > > > > Now, with the prototype of the refactored Pulsar CI build, the
> > > > resource
> > > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > > *This is about 60% reduction in resource consumption.* The
> whole
> > > > > pipeline
> > > > > > > completes in 75-100 minutes.
> > > > > > >
> > > > > > > Here's a breakdown of the duration (resource consumption) of
> each
> > > > build
> > > > > > job
> > > > > > > in the refactored workflow:
> > > > > > > Workflow Job seconds h:mm:ss
> > > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > > Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
> > > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849
> 0:14:09
> > > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
> > > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312
> 0:38:32
> > > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > > *Total resource consumption*
> > > > > > > 7:08:57
> > > > > > >
> > > > > > >
> > > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > > ).
> > > > > > > However, this is not a showstopper since there are ways to
> > address
> > > > the
> > > > > > > issues that cause flakiness.
> > > > > > > There is a separate PIP for changing the way to handle flaky
> > tests.
> > > > You
> > > > > > can
> > > > > > > find the link to that in the "Changes to GitHub Actions based
> > > Pulsar
> > > > > CI"
> > > > > > > document's header.
> > > > > > >
> > > > > > > *Some requests for the Pulsar community:*
> > > > > > >
> > > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > > . *It also contains more details of the prototype that has been
> > > > > > > successfully completed.*
> > > > > > >
> > > > > > > 2) *Please share your feedback and suggest a way forward.*
> > > > > > >
> > > > > > > *Thank you for your help!*
> > > > > > >
> > > > > > > BR, Lari
> > > > > > >
> > > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> > lari.hotari@sagire.fi
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Dear Pulsar community members,
> > > > > > > >
> > > > > > > > Currently, the Pulsar GitHub Actions workflows are consuming
> > the
> > > > > > majority
> > > > > > > > of the shared pool of resources allocated for
> > github.com/apache
> > > > > > > projects.
> > > > > > > > Other Apache projects have been impacted and there is a
> demand
> > to
> > > > > > improve
> > > > > > > > the Pulsar CI
> > > > > > > > <
> > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > >
> > > > > > > asap.
> > > > > > > >
> > > > > > > > In GitHub Actions Runners, the unit of resources is the time
> > > that a
> > > > > > > Runner
> > > > > > > > is occupied. I observed the workflow runs for handling a
> single
> > > > Pull
> > > > > > > > Request (in my personal fork) and these were the running
> > > durations:
> > > > > > > > Workflow name Duration
> > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > CI - Misc 0:16:51
> > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > CI - Unit 0:42:11
> > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > TOTAL 19:36:50
> > > > > > > >
> > > > > > > > *In this case, the total resource consumption of GitHub
> Actions
> > > > > Runners
> > > > > > > is
> > > > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > > > > apache/pulsar.*
> > > > > > > >
> > > > > > > > Since GitHub Actions Runner resource pool utilization is very
> > > high,
> > > > > > this
> > > > > > > > leads to the build queue to grow and take a long time to
> > process.
> > > > > > > >
> > > > > > > > I have been looking for ways to improve the Pulsar CI for the
> > > last
> > > > 3
> > > > > > > > months. During this period I worked on a few experiments. The
> > > > > learnings
> > > > > > > > from the past experiments are documented at a high level in
> the
> > > > > > following
> > > > > > > > draft PIP document.
> > > > > > > >
> > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> > > document
> > > > > is
> > > > > > a
> > > > > > > > Google doc:*
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > >
> > > > > > > > *Please participate* so that we get the plan adjusted based
> on
> > > the
> > > > > > > > feedback asap. If there's already a similar effort ongoing, I
> > > hope
> > > > we
> > > > > > can
> > > > > > > > join efforts.
> > > > > > > >
> > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > >
> > > > > > > > BR, Lari
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Sijie Guo <gu...@gmail.com>.
> *Sijie, how far are we from getting the draft PIP "Changes to GitHub
Actions based Pulsar CI" into an actual PIP that gets put on the wiki
page https://github.com/apache/pulsar/wiki
<https://github.com/apache/pulsar/wiki> ?*
I see what you referred to before now. This can be easily done. I (or any
other committer) can do it for you.
There is no real blocker for you to continue work even there are concerns
or it is not listed in the PIP.
We are using a lazy consensus approach. Typically if there is no objection,
please go ahead and not need to wait for approval.
If people raise concerns, please address the concerns.
> The reason why
re-runs happen currently is because of the high flakiness of tests.
Yes. So I am in favor of addressing flaky tests than merging all workflows
into one giant workflow.
It is not about "No pain, no gain". The community has suffered a lot with
giant workflow before.
There were a lot of committers and contributors working hard to split one
giant workflow into multiple
current workflows. Unless there is really strong evidence that merging them
back to one will improve
the entire CI experience, I will still have concerns about one giant
workflow approach.
Hence I am +1 to "changes to flaky test handing" and suggest focusing more
on solving flaky tests.
Consider merging them into one workflow when the tests are in a better
situation.
> This solution would also require disabling
required status checks
I don't think it is a good idea to disable status checks. We can consider
running "dark mode" but it will just overload the action quota.
Another alternative is to mirror the pull requests into another Github
account to test that and get more concrete statistics on the flakiness rate
of one workflow approach.
On Fri, Mar 12, 2021 at 1:57 AM Lari Hotari <La...@hotari.net> wrote:
> Thanks for the feedback, Sijie.
>
> > The "Fail fast" approach is great. That would be super helpful if there
> are
> > multiple workflows and each workflow is retryable.
> > However, I am not sure how much it will help if you run all workflows in
> > one giant workflow. Or is it making things worse?
>
> We can reduce the need for re-running workflow runs. The reason why
> re-runs happen currently is because of high flakiness of tests.
> Addressing flakiness continues to be top-priority. Now that the Pulsar CI
> workflow prototype is finished, I'll be focusing more in the other draft
> PIP, "Changes to flaky test handling".
> We as a community should address the critical problem that the current
> retry solution has: it can mask bugs in production code and make the build
> pass and allow changes to be merged that cause regressions.
> It's a false sense of security what the green builds after all the retries
> bring us. Bringing Pulsar to the next level in stability requires
> addressing this.
>
> If something doesn't work, it can be adapted and improved. Changes can be
> rolled back and revisited when things go worse. We also need a leap of
> faith.
> "No pain, no gain", like any change, it will be painful at first, but we
> will get over the bump.
>
> > Secondly, your test has been done in your folk where there are not a lot
> of
> > concurrent pushes and pull requests. I am not sure how your approach will
> > behave once it is merged into master. Can you simulate multiple
> concurrent
> > pull requests in your account to prove your approach doesn't bring side
> > effects?
>
> One possibility to address this is to introduce the new workflow in a mode
> where you need to opt-in to the new workflow in some way.
> This was an idea brought up by my colleagues Enrico and Andrey.
> It might be possible to configure the existing workflow and this new
> workflow in a way where some condition (for example whitelisted github user
> name or a certain keyword in the PR title/description) chooses either one
> for the Pull request. This solution would also require disabling
> required status checks ("Require status checks to pass before merging"
> feature in GitHub branch protection rules) since the names of the checks
> would be different.
>
> > Lastly, can we apply those optimizations to current workflows without
> > merging them into one giant workflow?
>
> This is what I have been doing. All individual optimizations have already
> been sent as PRs in the last months. I guess there's 20-30 PRs that have
> already been merged.
>
> https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
> There's also 2 build related PRs from yesterday which haven't been merged
> yet:
> Fix Maven download issues (ported from the prototype to our existing Pulsar
> CI): https://github.com/apache/pulsar/pull/9883
> improve Maven module build order (required for more efficient builds that
> selectively build required artifacts):
> https://github.com/apache/pulsar/pull/9882
>
> There aren't many optimizations left that could be ported from the
> prototype to the existing workflow. There are a few, but the impact is
> minor.
> The reason for this is that the optimization with the greatest impact are
> the ones that build a binary artifacts (maven libs, docker images) once and
> share it with the downstream jobs in the pipeline.
> Applying this type of solution has certain challenges when there are
> multiple separate workflow. Sharing binary artifacts to other workflows
> would require that the workflow to reuse the artifacts gets triggered by
> the workflow that produced the artifacts. This wouldn't be secure or
> practical for handling pull requests. Triggering a workflow explicitly
> would require a token from the main repository and using that for pull
> request builds would be a serious security vulnerability. (more details
> about the GitHub Actions security model in
>
> https://securitylab.github.com/research/github-actions-preventing-pwn-requests
> )
>
> *Sijie, how far are we from getting the draft PIP "Changes to GitHub
> Actions based Pulsar CI" into an actual PIP that gets put on the wiki
> page https://github.com/apache/pulsar/wiki
> <https://github.com/apache/pulsar/wiki> ?*
>
> -Lari
>
>
>
> On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com> wrote:
>
> > This is good progress. However, my main concern is still merging all
> > workflows into one giant workflow.
> >
> > The "Fail fast" approach is great. That would be super helpful if there
> are
> > multiple workflows and each workflow is retryable.
> > However, I am not sure how much it will help if you run all workflows in
> > one giant workflow. Or is it making things worse?
> >
> > Secondly, your test has been done in your folk where there are not a lot
> of
> > concurrent pushes and pull requests. I am not sure how your approach will
> > behave once it is merged into master. Can you simulate multiple
> concurrent
> > pull requests in your account to prove your approach doesn't bring side
> > effects?
> >
> > Lastly, can we apply those optimizations to current workflows without
> > merging them into one giant workflow?
> >
> > - Sijie
> >
> >
> > On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net> wrote:
> >
> > > Thanks for the feedback Michael.
> > >
> > > > I left a question on the doc about how concurrent runs affect the
> > > > repository's 5 GB cache limit.
> > >
> > > This is a good question. There isn't a clear answer in the GitHub
> Actions
> > > Cache documentation.
> > >
> > > The documentation is
> > >
> > >
> >
> https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > > .
> > > Based on this document and some testing, I have made these conclusions:
> > > For GitHub Actions Cache, pull requests get executed in the context of
> > the
> > > forked repository.
> > > The workflow triggered by a pull request event can only update it's own
> > > cache. It has read-only access to upstream caches.
> > > If there's a cache miss, the entry will get written to the cache of the
> > > forked repository. If the PR could to write to the upstream cache, it
> > would
> > > be a security issue since this would be vulnerable to cache poisoning
> > > attacks. Each repository has a 5GB quota for writes. The entries are
> kept
> > > up to 7 days.
> > > The performance is fairly good. Loading docker images from the
> repository
> > > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s. (the
> > > performance of the GHA cache is most likely higher since this is the
> > > throughput for docker load / docker save)
> > >
> > > If a single repository has a lot of concurrent jobs, it could start
> > > evicting caches.
> > > However that isn't likely to happen with the way Pulsar is developed
> > since
> > > pull requests are created from personal forks.
> > >
> > > > I also think it could be helpful to explicitly document, or reference
> > > > github documentation, on how failure will affect the DAG. I'm
> assuming
> > > that
> > > > if an action fails, its parallel peer actions will run to completion,
> > and
> > > > that the rest of the remaining stages will get canceled, but I
> haven't
> > > > worked with github actions before.
> > >
> > > For matrix jobs, "fail fast" is the default, which cancels all jobs in
> > the
> > > matrix if one fails. Other parallel flows would run to completion by
> > > default.
> > > In the prototype, I have added a Github script step to each job to
> cancel
> > > the complete workflow when a failure occurs.
> > > Here's an example:
> > >
> > >
> >
> https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> > >
> > > The prototype follows a "fail fast" design. When a failure occurs, fail
> > > fast and don't continue with other jobs.
> > > The benefit of this is that it reduces resource consumption. This helps
> > > keep the build queue short.
> > > When the build queue is short, developers get quick feedback from CI.
> > >
> > > Documenting all details in the PIP document isn't practical.
> > > *I'm hoping to start a separate document on low level details when
> there
> > is
> > > a high level acceptance of the proposed "Changes to GitHub Actions
> based
> > > Pulsar CI".*
> > > Together we can make this happen. We need decisions too. This proposal
> > > cannot stay as a draft forever.
> > > I'm looking forward to hearing from the Pulsar community, Pulsar
> > > committer and Pulsar PMC members how to take this forward.
> > >
> > > BR,
> > > -Lari
> > >
> > > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <
> mikemarsh17@gmail.com>
> > > wrote:
> > >
> > > > This will be a great improvement. I read through the PIP, and
> overall,
> > it
> > > > looks good to me.
> > > >
> > > > I left a question on the doc about how concurrent runs affect the
> > > > repository's 5 GB cache limit.
> > > >
> > > > I also think it could be helpful to explicitly document, or reference
> > > > github documentation, on how failure will affect the DAG. I'm
> assuming
> > > that
> > > > if an action fails, its parallel peer actions will run to completion,
> > and
> > > > that the rest of the remaining stages will get canceled, but I
> haven't
> > > > worked with github actions before.
> > > >
> > > > Thanks for all of the work you've put in so far.
> > > >
> > > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com> wrote:
> > > >
> > > > > This is great news. Thanks Hari , Mateo and pulsar community
> > > > >
> > > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <la...@sagire.fi>
> > > wrote:
> > > > >
> > > > > > Dear Pulsar community members,
> > > > > >
> > > > > > The work on "Changes to GitHub Actions based Pulsar CI" has gone
> > > > forward
> > > > > > based on your feedback. Here are some updates about the work.
> > > > > >
> > > > > > The draft PIP proposal document is here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > There's a *detailed status update in the document about a
> prototype
> > > for
> > > > > the
> > > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > > >
> > > > > > Thanks for all the suggestions and feedback by now. A lot of
> > > > improvements
> > > > > > have been made by the Pulsar contributors to overcome the
> technical
> > > > > > obstacles.
> > > > > > Special thanks go to Matteo for reducing the sizes of docker
> > images.
> > > A
> > > > > lot
> > > > > > of small improvements have been made to the Pulsar maven build to
> > > > enable
> > > > > > the new refactored GitHub Actions workflow. Thank you for all PR
> > > > reviews
> > > > > > and feedback.
> > > > > >
> > > > > > The main goal of the "Changes to GitHub Actions based Pulsar CI"
> > work
> > > > has
> > > > > > been to *reduce the resource consumption of the Pulsar CI build
> and
> > > to
> > > > > > speed up Pulsar development by improving the developer
> > productivity*
> > > > when
> > > > > > less time is wasted in waiting for Pulsar CI build feedback. The
> > > > > prototype
> > > > > > demonstrates these improvements.
> > > > > >
> > > > > > As you can see from the email from Jan 28 below, *the resource
> > > > > consumption
> > > > > > was 19 hrs 36 minutes* for a single pull request that was
> observed
> > > when
> > > > > the
> > > > > > work began.
> > > > > > Now, with the prototype of the refactored Pulsar CI build, the
> > > resource
> > > > > > consumption is *7 hrs 9 minutes.*
> > > > > > *This is about 60% reduction in resource consumption.* The whole
> > > > pipeline
> > > > > > completes in 75-100 minutes.
> > > > > >
> > > > > > Here's a breakdown of the duration (resource consumption) of each
> > > build
> > > > > job
> > > > > > in the refactored workflow:
> > > > > > Workflow Job seconds h:mm:ss
> > > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > > Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
> > > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > > Pulsar CI CI - Integration - Backwards Compatibility 849 0:14:09
> > > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
> > > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312 0:38:32
> > > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > > *Total resource consumption*
> > > > > > 7:08:57
> > > > > >
> > > > > >
> > > > > > GitHub Actions doesn't support restarting a single job (
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > > ).
> > > > > > However, this is not a showstopper since there are ways to
> address
> > > the
> > > > > > issues that cause flakiness.
> > > > > > There is a separate PIP for changing the way to handle flaky
> tests.
> > > You
> > > > > can
> > > > > > find the link to that in the "Changes to GitHub Actions based
> > Pulsar
> > > > CI"
> > > > > > document's header.
> > > > > >
> > > > > > *Some requests for the Pulsar community:*
> > > > > >
> > > > > > 1) *Please take a look at the updated PIP document*:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > > . *It also contains more details of the prototype that has been
> > > > > > successfully completed.*
> > > > > >
> > > > > > 2) *Please share your feedback and suggest a way forward.*
> > > > > >
> > > > > > *Thank you for your help!*
> > > > > >
> > > > > > BR, Lari
> > > > > >
> > > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <
> lari.hotari@sagire.fi
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Dear Pulsar community members,
> > > > > > >
> > > > > > > Currently, the Pulsar GitHub Actions workflows are consuming
> the
> > > > > majority
> > > > > > > of the shared pool of resources allocated for
> github.com/apache
> > > > > > projects.
> > > > > > > Other Apache projects have been impacted and there is a demand
> to
> > > > > improve
> > > > > > > the Pulsar CI
> > > > > > > <
> > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > >
> > > > > > asap.
> > > > > > >
> > > > > > > In GitHub Actions Runners, the unit of resources is the time
> > that a
> > > > > > Runner
> > > > > > > is occupied. I observed the workflow runs for handling a single
> > > Pull
> > > > > > > Request (in my personal fork) and these were the running
> > durations:
> > > > > > > Workflow name Duration
> > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > CI - Misc 0:16:51
> > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > CI - Unit 0:42:11
> > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > CI - Integration - Process 1:11:23
> > > > > > > CI - Shade - Test 1:15:45
> > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > TOTAL 19:36:50
> > > > > > >
> > > > > > > *In this case, the total resource consumption of GitHub Actions
> > > > Runners
> > > > > > is
> > > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > > > apache/pulsar.*
> > > > > > >
> > > > > > > Since GitHub Actions Runner resource pool utilization is very
> > high,
> > > > > this
> > > > > > > leads to the build queue to grow and take a long time to
> process.
> > > > > > >
> > > > > > > I have been looking for ways to improve the Pulsar CI for the
> > last
> > > 3
> > > > > > > months. During this period I worked on a few experiments. The
> > > > learnings
> > > > > > > from the past experiments are documented at a high level in the
> > > > > following
> > > > > > > draft PIP document.
> > > > > > >
> > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> > document
> > > > is
> > > > > a
> > > > > > > Google doc:*
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > >
> > > > > > > *Please participate* so that we get the plan adjusted based on
> > the
> > > > > > > feedback asap. If there's already a similar effort ongoing, I
> > hope
> > > we
> > > > > can
> > > > > > > join efforts.
> > > > > > >
> > > > > > > *Let's fix Pulsar CI!*
> > > > > > >
> > > > > > > BR, Lari
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Thanks for the feedback, Sijie.
> The "Fail fast" approach is great. That would be super helpful if there
are
> multiple workflows and each workflow is retryable.
> However, I am not sure how much it will help if you run all workflows in
> one giant workflow. Or is it making things worse?
We can reduce the need for re-running workflow runs. The reason why
re-runs happen currently is because of high flakiness of tests.
Addressing flakiness continues to be top-priority. Now that the Pulsar CI
workflow prototype is finished, I'll be focusing more in the other draft
PIP, "Changes to flaky test handling".
We as a community should address the critical problem that the current
retry solution has: it can mask bugs in production code and make the build
pass and allow changes to be merged that cause regressions.
It's a false sense of security what the green builds after all the retries
bring us. Bringing Pulsar to the next level in stability requires
addressing this.
If something doesn't work, it can be adapted and improved. Changes can be
rolled back and revisited when things go worse. We also need a leap of
faith.
"No pain, no gain", like any change, it will be painful at first, but we
will get over the bump.
> Secondly, your test has been done in your folk where there are not a lot
of
> concurrent pushes and pull requests. I am not sure how your approach will
> behave once it is merged into master. Can you simulate multiple concurrent
> pull requests in your account to prove your approach doesn't bring side
> effects?
One possibility to address this is to introduce the new workflow in a mode
where you need to opt-in to the new workflow in some way.
This was an idea brought up by my colleagues Enrico and Andrey.
It might be possible to configure the existing workflow and this new
workflow in a way where some condition (for example whitelisted github user
name or a certain keyword in the PR title/description) chooses either one
for the Pull request. This solution would also require disabling
required status checks ("Require status checks to pass before merging"
feature in GitHub branch protection rules) since the names of the checks
would be different.
> Lastly, can we apply those optimizations to current workflows without
> merging them into one giant workflow?
This is what I have been doing. All individual optimizations have already
been sent as PRs in the last months. I guess there's 20-30 PRs that have
already been merged.
https://github.com/apache/pulsar/pulls?q=is%3Apr+author%3Alhotari+is%3Amerged
There's also 2 build related PRs from yesterday which haven't been merged
yet:
Fix Maven download issues (ported from the prototype to our existing Pulsar
CI): https://github.com/apache/pulsar/pull/9883
improve Maven module build order (required for more efficient builds that
selectively build required artifacts):
https://github.com/apache/pulsar/pull/9882
There aren't many optimizations left that could be ported from the
prototype to the existing workflow. There are a few, but the impact is
minor.
The reason for this is that the optimization with the greatest impact are
the ones that build a binary artifacts (maven libs, docker images) once and
share it with the downstream jobs in the pipeline.
Applying this type of solution has certain challenges when there are
multiple separate workflow. Sharing binary artifacts to other workflows
would require that the workflow to reuse the artifacts gets triggered by
the workflow that produced the artifacts. This wouldn't be secure or
practical for handling pull requests. Triggering a workflow explicitly
would require a token from the main repository and using that for pull
request builds would be a serious security vulnerability. (more details
about the GitHub Actions security model in
https://securitylab.github.com/research/github-actions-preventing-pwn-requests
)
*Sijie, how far are we from getting the draft PIP "Changes to GitHub
Actions based Pulsar CI" into an actual PIP that gets put on the wiki
page https://github.com/apache/pulsar/wiki
<https://github.com/apache/pulsar/wiki> ?*
-Lari
On Fri, Mar 12, 2021 at 10:46 AM Sijie Guo <gu...@gmail.com> wrote:
> This is good progress. However, my main concern is still merging all
> workflows into one giant workflow.
>
> The "Fail fast" approach is great. That would be super helpful if there are
> multiple workflows and each workflow is retryable.
> However, I am not sure how much it will help if you run all workflows in
> one giant workflow. Or is it making things worse?
>
> Secondly, your test has been done in your folk where there are not a lot of
> concurrent pushes and pull requests. I am not sure how your approach will
> behave once it is merged into master. Can you simulate multiple concurrent
> pull requests in your account to prove your approach doesn't bring side
> effects?
>
> Lastly, can we apply those optimizations to current workflows without
> merging them into one giant workflow?
>
> - Sijie
>
>
> On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net> wrote:
>
> > Thanks for the feedback Michael.
> >
> > > I left a question on the doc about how concurrent runs affect the
> > > repository's 5 GB cache limit.
> >
> > This is a good question. There isn't a clear answer in the GitHub Actions
> > Cache documentation.
> >
> > The documentation is
> >
> >
> https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> > .
> > Based on this document and some testing, I have made these conclusions:
> > For GitHub Actions Cache, pull requests get executed in the context of
> the
> > forked repository.
> > The workflow triggered by a pull request event can only update it's own
> > cache. It has read-only access to upstream caches.
> > If there's a cache miss, the entry will get written to the cache of the
> > forked repository. If the PR could to write to the upstream cache, it
> would
> > be a security issue since this would be vulnerable to cache poisoning
> > attacks. Each repository has a 5GB quota for writes. The entries are kept
> > up to 7 days.
> > The performance is fairly good. Loading docker images from the repository
> > happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s. (the
> > performance of the GHA cache is most likely higher since this is the
> > throughput for docker load / docker save)
> >
> > If a single repository has a lot of concurrent jobs, it could start
> > evicting caches.
> > However that isn't likely to happen with the way Pulsar is developed
> since
> > pull requests are created from personal forks.
> >
> > > I also think it could be helpful to explicitly document, or reference
> > > github documentation, on how failure will affect the DAG. I'm assuming
> > that
> > > if an action fails, its parallel peer actions will run to completion,
> and
> > > that the rest of the remaining stages will get canceled, but I haven't
> > > worked with github actions before.
> >
> > For matrix jobs, "fail fast" is the default, which cancels all jobs in
> the
> > matrix if one fails. Other parallel flows would run to completion by
> > default.
> > In the prototype, I have added a Github script step to each job to cancel
> > the complete workflow when a failure occurs.
> > Here's an example:
> >
> >
> https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
> >
> > The prototype follows a "fail fast" design. When a failure occurs, fail
> > fast and don't continue with other jobs.
> > The benefit of this is that it reduces resource consumption. This helps
> > keep the build queue short.
> > When the build queue is short, developers get quick feedback from CI.
> >
> > Documenting all details in the PIP document isn't practical.
> > *I'm hoping to start a separate document on low level details when there
> is
> > a high level acceptance of the proposed "Changes to GitHub Actions based
> > Pulsar CI".*
> > Together we can make this happen. We need decisions too. This proposal
> > cannot stay as a draft forever.
> > I'm looking forward to hearing from the Pulsar community, Pulsar
> > committer and Pulsar PMC members how to take this forward.
> >
> > BR,
> > -Lari
> >
> > On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <mi...@gmail.com>
> > wrote:
> >
> > > This will be a great improvement. I read through the PIP, and overall,
> it
> > > looks good to me.
> > >
> > > I left a question on the doc about how concurrent runs affect the
> > > repository's 5 GB cache limit.
> > >
> > > I also think it could be helpful to explicitly document, or reference
> > > github documentation, on how failure will affect the DAG. I'm assuming
> > that
> > > if an action fails, its parallel peer actions will run to completion,
> and
> > > that the rest of the remaining stages will get canceled, but I haven't
> > > worked with github actions before.
> > >
> > > Thanks for all of the work you've put in so far.
> > >
> > > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com> wrote:
> > >
> > > > This is great news. Thanks Hari , Mateo and pulsar community
> > > >
> > > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <la...@sagire.fi>
> > wrote:
> > > >
> > > > > Dear Pulsar community members,
> > > > >
> > > > > The work on "Changes to GitHub Actions based Pulsar CI" has gone
> > > forward
> > > > > based on your feedback. Here are some updates about the work.
> > > > >
> > > > > The draft PIP proposal document is here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > There's a *detailed status update in the document about a prototype
> > for
> > > > the
> > > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > > >
> > > > > Thanks for all the suggestions and feedback by now. A lot of
> > > improvements
> > > > > have been made by the Pulsar contributors to overcome the technical
> > > > > obstacles.
> > > > > Special thanks go to Matteo for reducing the sizes of docker
> images.
> > A
> > > > lot
> > > > > of small improvements have been made to the Pulsar maven build to
> > > enable
> > > > > the new refactored GitHub Actions workflow. Thank you for all PR
> > > reviews
> > > > > and feedback.
> > > > >
> > > > > The main goal of the "Changes to GitHub Actions based Pulsar CI"
> work
> > > has
> > > > > been to *reduce the resource consumption of the Pulsar CI build and
> > to
> > > > > speed up Pulsar development by improving the developer
> productivity*
> > > when
> > > > > less time is wasted in waiting for Pulsar CI build feedback. The
> > > > prototype
> > > > > demonstrates these improvements.
> > > > >
> > > > > As you can see from the email from Jan 28 below, *the resource
> > > > consumption
> > > > > was 19 hrs 36 minutes* for a single pull request that was observed
> > when
> > > > the
> > > > > work began.
> > > > > Now, with the prototype of the refactored Pulsar CI build, the
> > resource
> > > > > consumption is *7 hrs 9 minutes.*
> > > > > *This is about 60% reduction in resource consumption.* The whole
> > > pipeline
> > > > > completes in 75-100 minutes.
> > > > >
> > > > > Here's a breakdown of the duration (resource consumption) of each
> > build
> > > > job
> > > > > in the refactored workflow:
> > > > > Workflow Job seconds h:mm:ss
> > > > > Pulsar CI Changed files check 4 0:00:04
> > > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > > Pulsar CI Build and License check 346 0:05:46
> > > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > > Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
> > > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > > Pulsar CI CI - Integration - Backwards Compatibility 849 0:14:09
> > > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
> > > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312 0:38:32
> > > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > > *Total resource consumption*
> > > > > 7:08:57
> > > > >
> > > > >
> > > > > GitHub Actions doesn't support restarting a single job (
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > > ).
> > > > > However, this is not a showstopper since there are ways to address
> > the
> > > > > issues that cause flakiness.
> > > > > There is a separate PIP for changing the way to handle flaky tests.
> > You
> > > > can
> > > > > find the link to that in the "Changes to GitHub Actions based
> Pulsar
> > > CI"
> > > > > document's header.
> > > > >
> > > > > *Some requests for the Pulsar community:*
> > > > >
> > > > > 1) *Please take a look at the updated PIP document*:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > > . *It also contains more details of the prototype that has been
> > > > > successfully completed.*
> > > > >
> > > > > 2) *Please share your feedback and suggest a way forward.*
> > > > >
> > > > > *Thank you for your help!*
> > > > >
> > > > > BR, Lari
> > > > >
> > > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <lari.hotari@sagire.fi
> >
> > > > wrote:
> > > > >
> > > > > > Dear Pulsar community members,
> > > > > >
> > > > > > Currently, the Pulsar GitHub Actions workflows are consuming the
> > > > majority
> > > > > > of the shared pool of resources allocated for github.com/apache
> > > > > projects.
> > > > > > Other Apache projects have been impacted and there is a demand to
> > > > improve
> > > > > > the Pulsar CI
> > > > > > <
> https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > >
> > > > > asap.
> > > > > >
> > > > > > In GitHub Actions Runners, the unit of resources is the time
> that a
> > > > > Runner
> > > > > > is occupied. I observed the workflow runs for handling a single
> > Pull
> > > > > > Request (in my personal fork) and these were the running
> durations:
> > > > > > Workflow name Duration
> > > > > > CI - Build - MacOS 0:17:23
> > > > > > CI - Go Functions style check 0:02:38
> > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > CI - Misc 0:16:51
> > > > > > CI - Unit - Proxy 0:14:23
> > > > > > CI - Go Functions Tests 0:22:08
> > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > CI - Unit 0:42:11
> > > > > > CI - Integration - Sql 1:00:13
> > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > CI - Integration - Function State 1:00:12
> > > > > > CI - Integration - Cli 1:10:22
> > > > > > CI - Integration - Transaction 1:16:34
> > > > > > CI - Integration - Process 1:11:23
> > > > > > CI - Shade - Test 1:15:45
> > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > CI - Integration - Standalone 0:45:29
> > > > > > CI - Integration - Messaging 1:00:23
> > > > > > CI - Integration - Thread 1:00:19
> > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > CI - Integration - Schema 1:00:19
> > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > TOTAL 19:36:50
> > > > > >
> > > > > > *In this case, the total resource consumption of GitHub Actions
> > > Runners
> > > > > is
> > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > > apache/pulsar.*
> > > > > >
> > > > > > Since GitHub Actions Runner resource pool utilization is very
> high,
> > > > this
> > > > > > leads to the build queue to grow and take a long time to process.
> > > > > >
> > > > > > I have been looking for ways to improve the Pulsar CI for the
> last
> > 3
> > > > > > months. During this period I worked on a few experiments. The
> > > learnings
> > > > > > from the past experiments are documented at a high level in the
> > > > following
> > > > > > draft PIP document.
> > > > > >
> > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> document
> > > is
> > > > a
> > > > > > Google doc:*
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > >
> > > > > > *Please participate* so that we get the plan adjusted based on
> the
> > > > > > feedback asap. If there's already a similar effort ongoing, I
> hope
> > we
> > > > can
> > > > > > join efforts.
> > > > > >
> > > > > > *Let's fix Pulsar CI!*
> > > > > >
> > > > > > BR, Lari
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Sijie Guo <gu...@gmail.com>.
This is good progress. However, my main concern is still merging all
workflows into one giant workflow.
The "Fail fast" approach is great. That would be super helpful if there are
multiple workflows and each workflow is retryable.
However, I am not sure how much it will help if you run all workflows in
one giant workflow. Or is it making things worse?
Secondly, your test has been done in your folk where there are not a lot of
concurrent pushes and pull requests. I am not sure how your approach will
behave once it is merged into master. Can you simulate multiple concurrent
pull requests in your account to prove your approach doesn't bring side
effects?
Lastly, can we apply those optimizations to current workflows without
merging them into one giant workflow?
- Sijie
On Fri, Mar 12, 2021 at 12:30 AM Lari Hotari <La...@hotari.net> wrote:
> Thanks for the feedback Michael.
>
> > I left a question on the doc about how concurrent runs affect the
> > repository's 5 GB cache limit.
>
> This is a good question. There isn't a clear answer in the GitHub Actions
> Cache documentation.
>
> The documentation is
>
> https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
> .
> Based on this document and some testing, I have made these conclusions:
> For GitHub Actions Cache, pull requests get executed in the context of the
> forked repository.
> The workflow triggered by a pull request event can only update it's own
> cache. It has read-only access to upstream caches.
> If there's a cache miss, the entry will get written to the cache of the
> forked repository. If the PR could to write to the upstream cache, it would
> be a security issue since this would be vulnerable to cache poisoning
> attacks. Each repository has a 5GB quota for writes. The entries are kept
> up to 7 days.
> The performance is fairly good. Loading docker images from the repository
> happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s. (the
> performance of the GHA cache is most likely higher since this is the
> throughput for docker load / docker save)
>
> If a single repository has a lot of concurrent jobs, it could start
> evicting caches.
> However that isn't likely to happen with the way Pulsar is developed since
> pull requests are created from personal forks.
>
> > I also think it could be helpful to explicitly document, or reference
> > github documentation, on how failure will affect the DAG. I'm assuming
> that
> > if an action fails, its parallel peer actions will run to completion, and
> > that the rest of the remaining stages will get canceled, but I haven't
> > worked with github actions before.
>
> For matrix jobs, "fail fast" is the default, which cancels all jobs in the
> matrix if one fails. Other parallel flows would run to completion by
> default.
> In the prototype, I have added a Github script step to each job to cancel
> the complete workflow when a failure occurs.
> Here's an example:
>
> https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
>
> The prototype follows a "fail fast" design. When a failure occurs, fail
> fast and don't continue with other jobs.
> The benefit of this is that it reduces resource consumption. This helps
> keep the build queue short.
> When the build queue is short, developers get quick feedback from CI.
>
> Documenting all details in the PIP document isn't practical.
> *I'm hoping to start a separate document on low level details when there is
> a high level acceptance of the proposed "Changes to GitHub Actions based
> Pulsar CI".*
> Together we can make this happen. We need decisions too. This proposal
> cannot stay as a draft forever.
> I'm looking forward to hearing from the Pulsar community, Pulsar
> committer and Pulsar PMC members how to take this forward.
>
> BR,
> -Lari
>
> On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <mi...@gmail.com>
> wrote:
>
> > This will be a great improvement. I read through the PIP, and overall, it
> > looks good to me.
> >
> > I left a question on the doc about how concurrent runs affect the
> > repository's 5 GB cache limit.
> >
> > I also think it could be helpful to explicitly document, or reference
> > github documentation, on how failure will affect the DAG. I'm assuming
> that
> > if an action fails, its parallel peer actions will run to completion, and
> > that the rest of the remaining stages will get canceled, but I haven't
> > worked with github actions before.
> >
> > Thanks for all of the work you've put in so far.
> >
> > On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com> wrote:
> >
> > > This is great news. Thanks Hari , Mateo and pulsar community
> > >
> > > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <la...@sagire.fi>
> wrote:
> > >
> > > > Dear Pulsar community members,
> > > >
> > > > The work on "Changes to GitHub Actions based Pulsar CI" has gone
> > forward
> > > > based on your feedback. Here are some updates about the work.
> > > >
> > > > The draft PIP proposal document is here:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > There's a *detailed status update in the document about a prototype
> for
> > > the
> > > > refactored Pulsar CI GitHub Actions based workflow*.
> > > >
> > > > Thanks for all the suggestions and feedback by now. A lot of
> > improvements
> > > > have been made by the Pulsar contributors to overcome the technical
> > > > obstacles.
> > > > Special thanks go to Matteo for reducing the sizes of docker images.
> A
> > > lot
> > > > of small improvements have been made to the Pulsar maven build to
> > enable
> > > > the new refactored GitHub Actions workflow. Thank you for all PR
> > reviews
> > > > and feedback.
> > > >
> > > > The main goal of the "Changes to GitHub Actions based Pulsar CI" work
> > has
> > > > been to *reduce the resource consumption of the Pulsar CI build and
> to
> > > > speed up Pulsar development by improving the developer productivity*
> > when
> > > > less time is wasted in waiting for Pulsar CI build feedback. The
> > > prototype
> > > > demonstrates these improvements.
> > > >
> > > > As you can see from the email from Jan 28 below, *the resource
> > > consumption
> > > > was 19 hrs 36 minutes* for a single pull request that was observed
> when
> > > the
> > > > work began.
> > > > Now, with the prototype of the refactored Pulsar CI build, the
> resource
> > > > consumption is *7 hrs 9 minutes.*
> > > > *This is about 60% reduction in resource consumption.* The whole
> > pipeline
> > > > completes in 75-100 minutes.
> > > >
> > > > Here's a breakdown of the duration (resource consumption) of each
> build
> > > job
> > > > in the refactored workflow:
> > > > Workflow Job seconds h:mm:ss
> > > > Pulsar CI Changed files check 4 0:00:04
> > > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > > Pulsar CI Build on MacOS 1017 0:16:57
> > > > Pulsar CI Build and License check 346 0:05:46
> > > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > > Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
> > > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > > Pulsar CI CI - Integration - Backwards Compatibility 849 0:14:09
> > > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > > Pulsar CI CI - System - Function State 699 0:11:39
> > > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
> > > > Pulsar CI CI - System - Pulsar Connectors - Process 2312 0:38:32
> > > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > > *Total resource consumption*
> > > > 7:08:57
> > > >
> > > >
> > > > GitHub Actions doesn't support restarting a single job (
> > > >
> > > >
> > >
> >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > > ).
> > > > However, this is not a showstopper since there are ways to address
> the
> > > > issues that cause flakiness.
> > > > There is a separate PIP for changing the way to handle flaky tests.
> You
> > > can
> > > > find the link to that in the "Changes to GitHub Actions based Pulsar
> > CI"
> > > > document's header.
> > > >
> > > > *Some requests for the Pulsar community:*
> > > >
> > > > 1) *Please take a look at the updated PIP document*:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > > . *It also contains more details of the prototype that has been
> > > > successfully completed.*
> > > >
> > > > 2) *Please share your feedback and suggest a way forward.*
> > > >
> > > > *Thank you for your help!*
> > > >
> > > > BR, Lari
> > > >
> > > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <la...@sagire.fi>
> > > wrote:
> > > >
> > > > > Dear Pulsar community members,
> > > > >
> > > > > Currently, the Pulsar GitHub Actions workflows are consuming the
> > > majority
> > > > > of the shared pool of resources allocated for github.com/apache
> > > > projects.
> > > > > Other Apache projects have been impacted and there is a demand to
> > > improve
> > > > > the Pulsar CI
> > > > > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> >
> > > > asap.
> > > > >
> > > > > In GitHub Actions Runners, the unit of resources is the time that a
> > > > Runner
> > > > > is occupied. I observed the workflow runs for handling a single
> Pull
> > > > > Request (in my personal fork) and these were the running durations:
> > > > > Workflow name Duration
> > > > > CI - Build - MacOS 0:17:23
> > > > > CI - Go Functions style check 0:02:38
> > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > CI - Misc 0:16:51
> > > > > CI - Unit - Proxy 0:14:23
> > > > > CI - Go Functions Tests 0:22:08
> > > > > CI - CPP, Python Tests 0:23:30
> > > > > CI - Unit 0:42:11
> > > > > CI - Integration - Sql 1:00:13
> > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > CI - Integration - Function State 1:00:12
> > > > > CI - Integration - Cli 1:10:22
> > > > > CI - Integration - Transaction 1:16:34
> > > > > CI - Integration - Process 1:11:23
> > > > > CI - Shade - Test 1:15:45
> > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > CI - Integration - Standalone 0:45:29
> > > > > CI - Integration - Messaging 1:00:23
> > > > > CI - Integration - Thread 1:00:19
> > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > CI - Integration - Schema 1:00:19
> > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > TOTAL 19:36:50
> > > > >
> > > > > *In this case, the total resource consumption of GitHub Actions
> > Runners
> > > > is
> > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > apache/pulsar.*
> > > > >
> > > > > Since GitHub Actions Runner resource pool utilization is very high,
> > > this
> > > > > leads to the build queue to grow and take a long time to process.
> > > > >
> > > > > I have been looking for ways to improve the Pulsar CI for the last
> 3
> > > > > months. During this period I worked on a few experiments. The
> > learnings
> > > > > from the past experiments are documented at a high level in the
> > > following
> > > > > draft PIP document.
> > > > >
> > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document
> > is
> > > a
> > > > > Google doc:*
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > >
> > > > > *Please participate* so that we get the plan adjusted based on the
> > > > > feedback asap. If there's already a similar effort ongoing, I hope
> we
> > > can
> > > > > join efforts.
> > > > >
> > > > > *Let's fix Pulsar CI!*
> > > > >
> > > > > BR, Lari
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Thanks for the feedback Michael.
> I left a question on the doc about how concurrent runs affect the
> repository's 5 GB cache limit.
This is a good question. There isn't a clear answer in the GitHub Actions
Cache documentation.
The documentation is
https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows
.
Based on this document and some testing, I have made these conclusions:
For GitHub Actions Cache, pull requests get executed in the context of the
forked repository.
The workflow triggered by a pull request event can only update it's own
cache. It has read-only access to upstream caches.
If there's a cache miss, the entry will get written to the cache of the
forked repository. If the PR could to write to the upstream cache, it would
be a security issue since this would be vulnerable to cache poisoning
attacks. Each repository has a 5GB quota for writes. The entries are kept
up to 7 days.
The performance is fairly good. Loading docker images from the repository
happens about 15MB/s. Writing is 2-3x slower, about 5-7MB/s. (the
performance of the GHA cache is most likely higher since this is the
throughput for docker load / docker save)
If a single repository has a lot of concurrent jobs, it could start
evicting caches.
However that isn't likely to happen with the way Pulsar is developed since
pull requests are created from personal forks.
> I also think it could be helpful to explicitly document, or reference
> github documentation, on how failure will affect the DAG. I'm assuming
that
> if an action fails, its parallel peer actions will run to completion, and
> that the rest of the remaining stages will get canceled, but I haven't
> worked with github actions before.
For matrix jobs, "fail fast" is the default, which cancels all jobs in the
matrix if one fails. Other parallel flows would run to completion by
default.
In the prototype, I have added a Github script step to each job to cancel
the complete workflow when a failure occurs.
Here's an example:
https://github.com/lhotari/pulsar/blob/lh-refactor-pulsar-ci-with-retries/.github/workflows/pulsar-ci.yaml#L281-L289
The prototype follows a "fail fast" design. When a failure occurs, fail
fast and don't continue with other jobs.
The benefit of this is that it reduces resource consumption. This helps
keep the build queue short.
When the build queue is short, developers get quick feedback from CI.
Documenting all details in the PIP document isn't practical.
*I'm hoping to start a separate document on low level details when there is
a high level acceptance of the proposed "Changes to GitHub Actions based
Pulsar CI".*
Together we can make this happen. We need decisions too. This proposal
cannot stay as a draft forever.
I'm looking forward to hearing from the Pulsar community, Pulsar
committer and Pulsar PMC members how to take this forward.
BR,
-Lari
On Fri, Mar 12, 2021 at 8:06 AM Michael Marshall <mi...@gmail.com>
wrote:
> This will be a great improvement. I read through the PIP, and overall, it
> looks good to me.
>
> I left a question on the doc about how concurrent runs affect the
> repository's 5 GB cache limit.
>
> I also think it could be helpful to explicitly document, or reference
> github documentation, on how failure will affect the DAG. I'm assuming that
> if an action fails, its parallel peer actions will run to completion, and
> that the rest of the remaining stages will get canceled, but I haven't
> worked with github actions before.
>
> Thanks for all of the work you've put in so far.
>
> On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com> wrote:
>
> > This is great news. Thanks Hari , Mateo and pulsar community
> >
> > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <la...@sagire.fi> wrote:
> >
> > > Dear Pulsar community members,
> > >
> > > The work on "Changes to GitHub Actions based Pulsar CI" has gone
> forward
> > > based on your feedback. Here are some updates about the work.
> > >
> > > The draft PIP proposal document is here:
> > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > There's a *detailed status update in the document about a prototype for
> > the
> > > refactored Pulsar CI GitHub Actions based workflow*.
> > >
> > > Thanks for all the suggestions and feedback by now. A lot of
> improvements
> > > have been made by the Pulsar contributors to overcome the technical
> > > obstacles.
> > > Special thanks go to Matteo for reducing the sizes of docker images. A
> > lot
> > > of small improvements have been made to the Pulsar maven build to
> enable
> > > the new refactored GitHub Actions workflow. Thank you for all PR
> reviews
> > > and feedback.
> > >
> > > The main goal of the "Changes to GitHub Actions based Pulsar CI" work
> has
> > > been to *reduce the resource consumption of the Pulsar CI build and to
> > > speed up Pulsar development by improving the developer productivity*
> when
> > > less time is wasted in waiting for Pulsar CI build feedback. The
> > prototype
> > > demonstrates these improvements.
> > >
> > > As you can see from the email from Jan 28 below, *the resource
> > consumption
> > > was 19 hrs 36 minutes* for a single pull request that was observed when
> > the
> > > work began.
> > > Now, with the prototype of the refactored Pulsar CI build, the resource
> > > consumption is *7 hrs 9 minutes.*
> > > *This is about 60% reduction in resource consumption.* The whole
> pipeline
> > > completes in 75-100 minutes.
> > >
> > > Here's a breakdown of the duration (resource consumption) of each build
> > job
> > > in the refactored workflow:
> > > Workflow Job seconds h:mm:ss
> > > Pulsar CI Changed files check 4 0:00:04
> > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > Pulsar CI Build on MacOS 1017 0:16:57
> > > Pulsar CI Build and License check 346 0:05:46
> > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
> > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > Pulsar CI CI - Integration - Backwards Compatibility 849 0:14:09
> > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > Pulsar CI CI - System - Function State 699 0:11:39
> > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
> > > Pulsar CI CI - System - Pulsar Connectors - Process 2312 0:38:32
> > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > *Total resource consumption*
> > > 7:08:57
> > >
> > >
> > > GitHub Actions doesn't support restarting a single job (
> > >
> > >
> >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > ).
> > > However, this is not a showstopper since there are ways to address the
> > > issues that cause flakiness.
> > > There is a separate PIP for changing the way to handle flaky tests. You
> > can
> > > find the link to that in the "Changes to GitHub Actions based Pulsar
> CI"
> > > document's header.
> > >
> > > *Some requests for the Pulsar community:*
> > >
> > > 1) *Please take a look at the updated PIP document*:
> > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > . *It also contains more details of the prototype that has been
> > > successfully completed.*
> > >
> > > 2) *Please share your feedback and suggest a way forward.*
> > >
> > > *Thank you for your help!*
> > >
> > > BR, Lari
> > >
> > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <la...@sagire.fi>
> > wrote:
> > >
> > > > Dear Pulsar community members,
> > > >
> > > > Currently, the Pulsar GitHub Actions workflows are consuming the
> > majority
> > > > of the shared pool of resources allocated for github.com/apache
> > > projects.
> > > > Other Apache projects have been impacted and there is a demand to
> > improve
> > > > the Pulsar CI
> > > > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396>
> > > asap.
> > > >
> > > > In GitHub Actions Runners, the unit of resources is the time that a
> > > Runner
> > > > is occupied. I observed the workflow runs for handling a single Pull
> > > > Request (in my personal fork) and these were the running durations:
> > > > Workflow name Duration
> > > > CI - Build - MacOS 0:17:23
> > > > CI - Go Functions style check 0:02:38
> > > > CI - Unit - Brokers - Other 0:15:40
> > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > CI - Misc 0:16:51
> > > > CI - Unit - Proxy 0:14:23
> > > > CI - Go Functions Tests 0:22:08
> > > > CI - CPP, Python Tests 0:23:30
> > > > CI - Unit 0:42:11
> > > > CI - Integration - Sql 1:00:13
> > > > CI - Integration - Tiered JCloud 1:00:18
> > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > CI - Integration - Function State 1:00:12
> > > > CI - Integration - Cli 1:10:22
> > > > CI - Integration - Transaction 1:16:34
> > > > CI - Integration - Process 1:11:23
> > > > CI - Shade - Test 1:15:45
> > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > CI - Integration - Standalone 0:45:29
> > > > CI - Integration - Messaging 1:00:23
> > > > CI - Integration - Thread 1:00:19
> > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > CI - Integration - Schema 1:00:19
> > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > TOTAL 19:36:50
> > > >
> > > > *In this case, the total resource consumption of GitHub Actions
> Runners
> > > is
> > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > apache/pulsar.*
> > > >
> > > > Since GitHub Actions Runner resource pool utilization is very high,
> > this
> > > > leads to the build queue to grow and take a long time to process.
> > > >
> > > > I have been looking for ways to improve the Pulsar CI for the last 3
> > > > months. During this period I worked on a few experiments. The
> learnings
> > > > from the past experiments are documented at a high level in the
> > following
> > > > draft PIP document.
> > > >
> > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document
> is
> > a
> > > > Google doc:*
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > >
> > > > *Please participate* so that we get the plan adjusted based on the
> > > > feedback asap. If there's already a similar effort ongoing, I hope we
> > can
> > > > join efforts.
> > > >
> > > > *Let's fix Pulsar CI!*
> > > >
> > > > BR, Lari
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Enrico Olivelli <eo...@gmail.com>.
Lari,
Thank you for this work.
I have been following it very closely and I am happy that we are close
to be able to leverage it very soon.
What is the next step in order to apply these changes to the Pulsar repo ?
IIUC all of the changes that blocked you on building up this work have
been merged (thanks also to Matteo)
I imagine that we can come with a PR that switches current CI
workflows to the new one.
Enrico
Il giorno ven 12 mar 2021 alle ore 07:06 Michael Marshall
<mi...@gmail.com> ha scritto:
>
> This will be a great improvement. I read through the PIP, and overall, it
> looks good to me.
>
> I left a question on the doc about how concurrent runs affect the
> repository's 5 GB cache limit.
>
> I also think it could be helpful to explicitly document, or reference
> github documentation, on how failure will affect the DAG. I'm assuming that
> if an action fails, its parallel peer actions will run to completion, and
> that the rest of the remaining stages will get canceled, but I haven't
> worked with github actions before.
>
> Thanks for all of the work you've put in so far.
>
> On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com> wrote:
>
> > This is great news. Thanks Hari , Mateo and pulsar community
> >
> > On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <la...@sagire.fi> wrote:
> >
> > > Dear Pulsar community members,
> > >
> > > The work on "Changes to GitHub Actions based Pulsar CI" has gone forward
> > > based on your feedback. Here are some updates about the work.
> > >
> > > The draft PIP proposal document is here:
> > >
> > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > There's a *detailed status update in the document about a prototype for
> > the
> > > refactored Pulsar CI GitHub Actions based workflow*.
> > >
> > > Thanks for all the suggestions and feedback by now. A lot of improvements
> > > have been made by the Pulsar contributors to overcome the technical
> > > obstacles.
> > > Special thanks go to Matteo for reducing the sizes of docker images. A
> > lot
> > > of small improvements have been made to the Pulsar maven build to enable
> > > the new refactored GitHub Actions workflow. Thank you for all PR reviews
> > > and feedback.
> > >
> > > The main goal of the "Changes to GitHub Actions based Pulsar CI" work has
> > > been to *reduce the resource consumption of the Pulsar CI build and to
> > > speed up Pulsar development by improving the developer productivity* when
> > > less time is wasted in waiting for Pulsar CI build feedback. The
> > prototype
> > > demonstrates these improvements.
> > >
> > > As you can see from the email from Jan 28 below, *the resource
> > consumption
> > > was 19 hrs 36 minutes* for a single pull request that was observed when
> > the
> > > work began.
> > > Now, with the prototype of the refactored Pulsar CI build, the resource
> > > consumption is *7 hrs 9 minutes.*
> > > *This is about 60% reduction in resource consumption.* The whole pipeline
> > > completes in 75-100 minutes.
> > >
> > > Here's a breakdown of the duration (resource consumption) of each build
> > job
> > > in the refactored workflow:
> > > Workflow Job seconds h:mm:ss
> > > Pulsar CI Changed files check 4 0:00:04
> > > Pulsar CI Go 1.11 Functions 155 0:02:35
> > > Pulsar CI Go 1.12 Functions 166 0:02:46
> > > Pulsar CI Go 1.13 Functions 113 0:01:53
> > > Pulsar CI Go 1.14 Functions 96 0:01:36
> > > Pulsar CI Build on MacOS 1017 0:16:57
> > > Pulsar CI Build and License check 346 0:05:46
> > > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > > Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
> > > Pulsar CI CI - Unit - Other 1580 0:26:20
> > > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > > Pulsar CI CI - Integration - Shade 414 0:06:54
> > > Pulsar CI CI - Integration - Backwards Compatibility 849 0:14:09
> > > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > > Pulsar CI CI - Integration - Schema 468 0:07:48
> > > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > > Pulsar CI CI - System - Function State 699 0:11:39
> > > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > > Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
> > > Pulsar CI CI - System - Pulsar Connectors - Process 2312 0:38:32
> > > Pulsar CI CI - System - Sql 1377 0:22:57
> > > *Total resource consumption*
> > > 7:08:57
> > >
> > >
> > > GitHub Actions doesn't support restarting a single job (
> > >
> > >
> > https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > > ).
> > > However, this is not a showstopper since there are ways to address the
> > > issues that cause flakiness.
> > > There is a separate PIP for changing the way to handle flaky tests. You
> > can
> > > find the link to that in the "Changes to GitHub Actions based Pulsar CI"
> > > document's header.
> > >
> > > *Some requests for the Pulsar community:*
> > >
> > > 1) *Please take a look at the updated PIP document*:
> > >
> > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > > . *It also contains more details of the prototype that has been
> > > successfully completed.*
> > >
> > > 2) *Please share your feedback and suggest a way forward.*
> > >
> > > *Thank you for your help!*
> > >
> > > BR, Lari
> > >
> > > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <la...@sagire.fi>
> > wrote:
> > >
> > > > Dear Pulsar community members,
> > > >
> > > > Currently, the Pulsar GitHub Actions workflows are consuming the
> > majority
> > > > of the shared pool of resources allocated for github.com/apache
> > > projects.
> > > > Other Apache projects have been impacted and there is a demand to
> > improve
> > > > the Pulsar CI
> > > > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396>
> > > asap.
> > > >
> > > > In GitHub Actions Runners, the unit of resources is the time that a
> > > Runner
> > > > is occupied. I observed the workflow runs for handling a single Pull
> > > > Request (in my personal fork) and these were the running durations:
> > > > Workflow name Duration
> > > > CI - Build - MacOS 0:17:23
> > > > CI - Go Functions style check 0:02:38
> > > > CI - Unit - Brokers - Other 0:15:40
> > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > CI - Misc 0:16:51
> > > > CI - Unit - Proxy 0:14:23
> > > > CI - Go Functions Tests 0:22:08
> > > > CI - CPP, Python Tests 0:23:30
> > > > CI - Unit 0:42:11
> > > > CI - Integration - Sql 1:00:13
> > > > CI - Integration - Tiered JCloud 1:00:18
> > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > CI - Integration - Function State 1:00:12
> > > > CI - Integration - Cli 1:10:22
> > > > CI - Integration - Transaction 1:16:34
> > > > CI - Integration - Process 1:11:23
> > > > CI - Shade - Test 1:15:45
> > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > CI - Integration - Standalone 0:45:29
> > > > CI - Integration - Messaging 1:00:23
> > > > CI - Integration - Thread 1:00:19
> > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > CI - Integration - Schema 1:00:19
> > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > TOTAL 19:36:50
> > > >
> > > > *In this case, the total resource consumption of GitHub Actions Runners
> > > is
> > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > apache/pulsar.*
> > > >
> > > > Since GitHub Actions Runner resource pool utilization is very high,
> > this
> > > > leads to the build queue to grow and take a long time to process.
> > > >
> > > > I have been looking for ways to improve the Pulsar CI for the last 3
> > > > months. During this period I worked on a few experiments. The learnings
> > > > from the past experiments are documented at a high level in the
> > following
> > > > draft PIP document.
> > > >
> > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document is
> > a
> > > > Google doc:*
> > > >
> > > >
> > >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > >
> > > > *Please participate* so that we get the plan adjusted based on the
> > > > feedback asap. If there's already a similar effort ongoing, I hope we
> > can
> > > > join efforts.
> > > >
> > > > *Let's fix Pulsar CI!*
> > > >
> > > > BR, Lari
> > > >
> > >
> >
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Michael Marshall <mi...@gmail.com>.
This will be a great improvement. I read through the PIP, and overall, it
looks good to me.
I left a question on the doc about how concurrent runs affect the
repository's 5 GB cache limit.
I also think it could be helpful to explicitly document, or reference
github documentation, on how failure will affect the DAG. I'm assuming that
if an action fails, its parallel peer actions will run to completion, and
that the rest of the remaining stages will get canceled, but I haven't
worked with github actions before.
Thanks for all of the work you've put in so far.
On Thu, Mar 11, 2021 at 6:37 PM Yuva raj <uv...@gmail.com> wrote:
> This is great news. Thanks Hari , Mateo and pulsar community
>
> On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <la...@sagire.fi> wrote:
>
> > Dear Pulsar community members,
> >
> > The work on "Changes to GitHub Actions based Pulsar CI" has gone forward
> > based on your feedback. Here are some updates about the work.
> >
> > The draft PIP proposal document is here:
> >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > There's a *detailed status update in the document about a prototype for
> the
> > refactored Pulsar CI GitHub Actions based workflow*.
> >
> > Thanks for all the suggestions and feedback by now. A lot of improvements
> > have been made by the Pulsar contributors to overcome the technical
> > obstacles.
> > Special thanks go to Matteo for reducing the sizes of docker images. A
> lot
> > of small improvements have been made to the Pulsar maven build to enable
> > the new refactored GitHub Actions workflow. Thank you for all PR reviews
> > and feedback.
> >
> > The main goal of the "Changes to GitHub Actions based Pulsar CI" work has
> > been to *reduce the resource consumption of the Pulsar CI build and to
> > speed up Pulsar development by improving the developer productivity* when
> > less time is wasted in waiting for Pulsar CI build feedback. The
> prototype
> > demonstrates these improvements.
> >
> > As you can see from the email from Jan 28 below, *the resource
> consumption
> > was 19 hrs 36 minutes* for a single pull request that was observed when
> the
> > work began.
> > Now, with the prototype of the refactored Pulsar CI build, the resource
> > consumption is *7 hrs 9 minutes.*
> > *This is about 60% reduction in resource consumption.* The whole pipeline
> > completes in 75-100 minutes.
> >
> > Here's a breakdown of the duration (resource consumption) of each build
> job
> > in the refactored workflow:
> > Workflow Job seconds h:mm:ss
> > Pulsar CI Changed files check 4 0:00:04
> > Pulsar CI Go 1.11 Functions 155 0:02:35
> > Pulsar CI Go 1.12 Functions 166 0:02:46
> > Pulsar CI Go 1.13 Functions 113 0:01:53
> > Pulsar CI Go 1.14 Functions 96 0:01:36
> > Pulsar CI Build on MacOS 1017 0:16:57
> > Pulsar CI Build and License check 346 0:05:46
> > Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> > Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
> > Pulsar CI CI - Unit - Other 1580 0:26:20
> > Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> > Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> > Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> > Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> > Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> > Pulsar CI CI - Unit - Proxy 331 0:05:31
> > Pulsar CI Build Pulsar docker image 2343 0:39:03
> > Pulsar CI CI - Integration - Shade 414 0:06:54
> > Pulsar CI CI - Integration - Backwards Compatibility 849 0:14:09
> > Pulsar CI CI - Integration - Cli 1490 0:24:50
> > Pulsar CI CI - Integration - Messaging 857 0:14:17
> > Pulsar CI CI - Integration - Schema 468 0:07:48
> > Pulsar CI CI - Integration - Standalone 286 0:04:46
> > Pulsar CI CI - Integration - Transaction 362 0:06:02
> > Pulsar CI CI - System - Function State 699 0:11:39
> > Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> > Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> > Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
> > Pulsar CI CI - System - Pulsar Connectors - Process 2312 0:38:32
> > Pulsar CI CI - System - Sql 1377 0:22:57
> > *Total resource consumption*
> > 7:08:57
> >
> >
> > GitHub Actions doesn't support restarting a single job (
> >
> >
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> > ).
> > However, this is not a showstopper since there are ways to address the
> > issues that cause flakiness.
> > There is a separate PIP for changing the way to handle flaky tests. You
> can
> > find the link to that in the "Changes to GitHub Actions based Pulsar CI"
> > document's header.
> >
> > *Some requests for the Pulsar community:*
> >
> > 1) *Please take a look at the updated PIP document*:
> >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> > . *It also contains more details of the prototype that has been
> > successfully completed.*
> >
> > 2) *Please share your feedback and suggest a way forward.*
> >
> > *Thank you for your help!*
> >
> > BR, Lari
> >
> > On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <la...@sagire.fi>
> wrote:
> >
> > > Dear Pulsar community members,
> > >
> > > Currently, the Pulsar GitHub Actions workflows are consuming the
> majority
> > > of the shared pool of resources allocated for github.com/apache
> > projects.
> > > Other Apache projects have been impacted and there is a demand to
> improve
> > > the Pulsar CI
> > > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396>
> > asap.
> > >
> > > In GitHub Actions Runners, the unit of resources is the time that a
> > Runner
> > > is occupied. I observed the workflow runs for handling a single Pull
> > > Request (in my personal fork) and these were the running durations:
> > > Workflow name Duration
> > > CI - Build - MacOS 0:17:23
> > > CI - Go Functions style check 0:02:38
> > > CI - Unit - Brokers - Other 0:15:40
> > > CI - Unit - Brokers - Client Impl 0:16:28
> > > CI - Misc 0:16:51
> > > CI - Unit - Proxy 0:14:23
> > > CI - Go Functions Tests 0:22:08
> > > CI - CPP, Python Tests 0:23:30
> > > CI - Unit 0:42:11
> > > CI - Integration - Sql 1:00:13
> > > CI - Integration - Tiered JCloud 1:00:18
> > > CI - Integration - Tiered FileSystem 1:00:13
> > > CI - Integration - Function State 1:00:12
> > > CI - Integration - Cli 1:10:22
> > > CI - Integration - Transaction 1:16:34
> > > CI - Integration - Process 1:11:23
> > > CI - Shade - Test 1:15:45
> > > CI - Unit - Brokers - Client Api 0:26:13
> > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > CI - Integration - Standalone 0:45:29
> > > CI - Integration - Messaging 1:00:23
> > > CI - Integration - Thread 1:00:19
> > > CI - Integration - Backwards Compatibility 1:00:19
> > > CI - Integration - Schema 1:00:19
> > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > TOTAL 19:36:50
> > >
> > > *In this case, the total resource consumption of GitHub Actions Runners
> > is
> > > 19 hours 36 minutes 50 seconds for a single pull request to
> > apache/pulsar.*
> > >
> > > Since GitHub Actions Runner resource pool utilization is very high,
> this
> > > leads to the build queue to grow and take a long time to process.
> > >
> > > I have been looking for ways to improve the Pulsar CI for the last 3
> > > months. During this period I worked on a few experiments. The learnings
> > > from the past experiments are documented at a high level in the
> following
> > > draft PIP document.
> > >
> > > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document is
> a
> > > Google doc:*
> > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > >
> > > *Please participate* so that we get the plan adjusted based on the
> > > feedback asap. If there's already a similar effort ongoing, I hope we
> can
> > > join efforts.
> > >
> > > *Let's fix Pulsar CI!*
> > >
> > > BR, Lari
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Yuva raj <uv...@gmail.com>.
This is great news. Thanks Hari , Mateo and pulsar community
On Fri, Mar 12, 2021, 2:04 AM Lari Hotari <la...@sagire.fi> wrote:
> Dear Pulsar community members,
>
> The work on "Changes to GitHub Actions based Pulsar CI" has gone forward
> based on your feedback. Here are some updates about the work.
>
> The draft PIP proposal document is here:
>
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> There's a *detailed status update in the document about a prototype for the
> refactored Pulsar CI GitHub Actions based workflow*.
>
> Thanks for all the suggestions and feedback by now. A lot of improvements
> have been made by the Pulsar contributors to overcome the technical
> obstacles.
> Special thanks go to Matteo for reducing the sizes of docker images. A lot
> of small improvements have been made to the Pulsar maven build to enable
> the new refactored GitHub Actions workflow. Thank you for all PR reviews
> and feedback.
>
> The main goal of the "Changes to GitHub Actions based Pulsar CI" work has
> been to *reduce the resource consumption of the Pulsar CI build and to
> speed up Pulsar development by improving the developer productivity* when
> less time is wasted in waiting for Pulsar CI build feedback. The prototype
> demonstrates these improvements.
>
> As you can see from the email from Jan 28 below, *the resource consumption
> was 19 hrs 36 minutes* for a single pull request that was observed when the
> work began.
> Now, with the prototype of the refactored Pulsar CI build, the resource
> consumption is *7 hrs 9 minutes.*
> *This is about 60% reduction in resource consumption.* The whole pipeline
> completes in 75-100 minutes.
>
> Here's a breakdown of the duration (resource consumption) of each build job
> in the refactored workflow:
> Workflow Job seconds h:mm:ss
> Pulsar CI Changed files check 4 0:00:04
> Pulsar CI Go 1.11 Functions 155 0:02:35
> Pulsar CI Go 1.12 Functions 166 0:02:46
> Pulsar CI Go 1.13 Functions 113 0:01:53
> Pulsar CI Go 1.14 Functions 96 0:01:36
> Pulsar CI Build on MacOS 1017 0:16:57
> Pulsar CI Build and License check 346 0:05:46
> Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
> Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
> Pulsar CI CI - Unit - Other 1580 0:26:20
> Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
> Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
> Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
> Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
> Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
> Pulsar CI CI - Unit - Proxy 331 0:05:31
> Pulsar CI Build Pulsar docker image 2343 0:39:03
> Pulsar CI CI - Integration - Shade 414 0:06:54
> Pulsar CI CI - Integration - Backwards Compatibility 849 0:14:09
> Pulsar CI CI - Integration - Cli 1490 0:24:50
> Pulsar CI CI - Integration - Messaging 857 0:14:17
> Pulsar CI CI - Integration - Schema 468 0:07:48
> Pulsar CI CI - Integration - Standalone 286 0:04:46
> Pulsar CI CI - Integration - Transaction 362 0:06:02
> Pulsar CI CI - System - Function State 699 0:11:39
> Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
> Pulsar CI CI - System - Tiered JCloud 529 0:08:49
> Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
> Pulsar CI CI - System - Pulsar Connectors - Process 2312 0:38:32
> Pulsar CI CI - System - Sql 1377 0:22:57
> *Total resource consumption*
> 7:08:57
>
>
> GitHub Actions doesn't support restarting a single job (
>
> https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
> ).
> However, this is not a showstopper since there are ways to address the
> issues that cause flakiness.
> There is a separate PIP for changing the way to handle flaky tests. You can
> find the link to that in the "Changes to GitHub Actions based Pulsar CI"
> document's header.
>
> *Some requests for the Pulsar community:*
>
> 1) *Please take a look at the updated PIP document*:
>
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
> . *It also contains more details of the prototype that has been
> successfully completed.*
>
> 2) *Please share your feedback and suggest a way forward.*
>
> *Thank you for your help!*
>
> BR, Lari
>
> On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <la...@sagire.fi> wrote:
>
> > Dear Pulsar community members,
> >
> > Currently, the Pulsar GitHub Actions workflows are consuming the majority
> > of the shared pool of resources allocated for github.com/apache
> projects.
> > Other Apache projects have been impacted and there is a demand to improve
> > the Pulsar CI
> > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396>
> asap.
> >
> > In GitHub Actions Runners, the unit of resources is the time that a
> Runner
> > is occupied. I observed the workflow runs for handling a single Pull
> > Request (in my personal fork) and these were the running durations:
> > Workflow name Duration
> > CI - Build - MacOS 0:17:23
> > CI - Go Functions style check 0:02:38
> > CI - Unit - Brokers - Other 0:15:40
> > CI - Unit - Brokers - Client Impl 0:16:28
> > CI - Misc 0:16:51
> > CI - Unit - Proxy 0:14:23
> > CI - Go Functions Tests 0:22:08
> > CI - CPP, Python Tests 0:23:30
> > CI - Unit 0:42:11
> > CI - Integration - Sql 1:00:13
> > CI - Integration - Tiered JCloud 1:00:18
> > CI - Integration - Tiered FileSystem 1:00:13
> > CI - Integration - Function State 1:00:12
> > CI - Integration - Cli 1:10:22
> > CI - Integration - Transaction 1:16:34
> > CI - Integration - Process 1:11:23
> > CI - Shade - Test 1:15:45
> > CI - Unit - Brokers - Client Api 0:26:13
> > CI - Unit - Brokers - Broker Group 2 0:35:05
> > CI - Integration - Standalone 0:45:29
> > CI - Integration - Messaging 1:00:23
> > CI - Integration - Thread 1:00:19
> > CI - Integration - Backwards Compatibility 1:00:19
> > CI - Integration - Schema 1:00:19
> > CI - Unit - Brokers - Broker Group 1 2:02:31
> > TOTAL 19:36:50
> >
> > *In this case, the total resource consumption of GitHub Actions Runners
> is
> > 19 hours 36 minutes 50 seconds for a single pull request to
> apache/pulsar.*
> >
> > Since GitHub Actions Runner resource pool utilization is very high, this
> > leads to the build queue to grow and take a long time to process.
> >
> > I have been looking for ways to improve the Pulsar CI for the last 3
> > months. During this period I worked on a few experiments. The learnings
> > from the past experiments are documented at a high level in the following
> > draft PIP document.
> >
> > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document is a
> > Google doc:*
> >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> >
> > *Please participate* so that we get the plan adjusted based on the
> > feedback asap. If there's already a similar effort ongoing, I hope we can
> > join efforts.
> >
> > *Let's fix Pulsar CI!*
> >
> > BR, Lari
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <la...@sagire.fi>.
Dear Pulsar community members,
The work on "Changes to GitHub Actions based Pulsar CI" has gone forward
based on your feedback. Here are some updates about the work.
The draft PIP proposal document is here:
https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
There's a *detailed status update in the document about a prototype for the
refactored Pulsar CI GitHub Actions based workflow*.
Thanks for all the suggestions and feedback by now. A lot of improvements
have been made by the Pulsar contributors to overcome the technical
obstacles.
Special thanks go to Matteo for reducing the sizes of docker images. A lot
of small improvements have been made to the Pulsar maven build to enable
the new refactored GitHub Actions workflow. Thank you for all PR reviews
and feedback.
The main goal of the "Changes to GitHub Actions based Pulsar CI" work has
been to *reduce the resource consumption of the Pulsar CI build and to
speed up Pulsar development by improving the developer productivity* when
less time is wasted in waiting for Pulsar CI build feedback. The prototype
demonstrates these improvements.
As you can see from the email from Jan 28 below, *the resource consumption
was 19 hrs 36 minutes* for a single pull request that was observed when the
work began.
Now, with the prototype of the refactored Pulsar CI build, the resource
consumption is *7 hrs 9 minutes.*
*This is about 60% reduction in resource consumption.* The whole pipeline
completes in 75-100 minutes.
Here's a breakdown of the duration (resource consumption) of each build job
in the refactored workflow:
Workflow Job seconds h:mm:ss
Pulsar CI Changed files check 4 0:00:04
Pulsar CI Go 1.11 Functions 155 0:02:35
Pulsar CI Go 1.12 Functions 166 0:02:46
Pulsar CI Go 1.13 Functions 113 0:01:53
Pulsar CI Go 1.14 Functions 96 0:01:36
Pulsar CI Build on MacOS 1017 0:16:57
Pulsar CI Build and License check 346 0:05:46
Pulsar CI Build Pulsar CPP and Python clients 683 0:11:23
Pulsar CI Build Pulsar java-test-image docker image 405 0:06:45
Pulsar CI CI - Unit - Other 1580 0:26:20
Pulsar CI CI - Unit - Brokers - Broker Group 1 968 0:16:08
Pulsar CI CI - Unit - Brokers - Broker Group 2 2223 0:37:03
Pulsar CI CI - Unit - Brokers - Client Api 1652 0:27:32
Pulsar CI CI - Unit - Brokers - Client Impl 916 0:15:16
Pulsar CI CI - Unit - Brokers - Other 522 0:08:42
Pulsar CI CI - Unit - Proxy 331 0:05:31
Pulsar CI Build Pulsar docker image 2343 0:39:03
Pulsar CI CI - Integration - Shade 414 0:06:54
Pulsar CI CI - Integration - Backwards Compatibility 849 0:14:09
Pulsar CI CI - Integration - Cli 1490 0:24:50
Pulsar CI CI - Integration - Messaging 857 0:14:17
Pulsar CI CI - Integration - Schema 468 0:07:48
Pulsar CI CI - Integration - Standalone 286 0:04:46
Pulsar CI CI - Integration - Transaction 362 0:06:02
Pulsar CI CI - System - Function State 699 0:11:39
Pulsar CI CI - System - Tiered FileSystem 779 0:12:59
Pulsar CI CI - System - Tiered JCloud 529 0:08:49
Pulsar CI CI - System - Pulsar Connectors - Thread 1795 0:29:55
Pulsar CI CI - System - Pulsar Connectors - Process 2312 0:38:32
Pulsar CI CI - System - Sql 1377 0:22:57
*Total resource consumption*
7:08:57
GitHub Actions doesn't support restarting a single job (
https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234
).
However, this is not a showstopper since there are ways to address the
issues that cause flakiness.
There is a separate PIP for changing the way to handle flaky tests. You can
find the link to that in the "Changes to GitHub Actions based Pulsar CI"
document's header.
*Some requests for the Pulsar community:*
1) *Please take a look at the updated PIP document*:
https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit#heading=h.f53rkcu20sry
. *It also contains more details of the prototype that has been
successfully completed.*
2) *Please share your feedback and suggest a way forward.*
*Thank you for your help!*
BR, Lari
On Thu, Jan 28, 2021 at 7:13 PM Lari Hotari <la...@sagire.fi> wrote:
> Dear Pulsar community members,
>
> Currently, the Pulsar GitHub Actions workflows are consuming the majority
> of the shared pool of resources allocated for github.com/apache projects.
> Other Apache projects have been impacted and there is a demand to improve
> the Pulsar CI
> <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396> asap.
>
> In GitHub Actions Runners, the unit of resources is the time that a Runner
> is occupied. I observed the workflow runs for handling a single Pull
> Request (in my personal fork) and these were the running durations:
> Workflow name Duration
> CI - Build - MacOS 0:17:23
> CI - Go Functions style check 0:02:38
> CI - Unit - Brokers - Other 0:15:40
> CI - Unit - Brokers - Client Impl 0:16:28
> CI - Misc 0:16:51
> CI - Unit - Proxy 0:14:23
> CI - Go Functions Tests 0:22:08
> CI - CPP, Python Tests 0:23:30
> CI - Unit 0:42:11
> CI - Integration - Sql 1:00:13
> CI - Integration - Tiered JCloud 1:00:18
> CI - Integration - Tiered FileSystem 1:00:13
> CI - Integration - Function State 1:00:12
> CI - Integration - Cli 1:10:22
> CI - Integration - Transaction 1:16:34
> CI - Integration - Process 1:11:23
> CI - Shade - Test 1:15:45
> CI - Unit - Brokers - Client Api 0:26:13
> CI - Unit - Brokers - Broker Group 2 0:35:05
> CI - Integration - Standalone 0:45:29
> CI - Integration - Messaging 1:00:23
> CI - Integration - Thread 1:00:19
> CI - Integration - Backwards Compatibility 1:00:19
> CI - Integration - Schema 1:00:19
> CI - Unit - Brokers - Broker Group 1 2:02:31
> TOTAL 19:36:50
>
> *In this case, the total resource consumption of GitHub Actions Runners is
> 19 hours 36 minutes 50 seconds for a single pull request to apache/pulsar.*
>
> Since GitHub Actions Runner resource pool utilization is very high, this
> leads to the build queue to grow and take a long time to process.
>
> I have been looking for ways to improve the Pulsar CI for the last 3
> months. During this period I worked on a few experiments. The learnings
> from the past experiments are documented at a high level in the following
> draft PIP document.
>
> *The draft PIP "Changes to GitHub Actions based Pulsar CI" document is a
> Google doc:*
>
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
>
> *Please participate* so that we get the plan adjusted based on the
> feedback asap. If there's already a similar effort ongoing, I hope we can
> join efforts.
>
> *Let's fix Pulsar CI!*
>
> BR, Lari
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Hi Sijie,
> Yes. I was doing the work there and hopefully will get there soon.
Is the plan to do the 2.8.0 release with the new repositories and
repository structure in place?
I'd be interested in understanding how releasing is intended to be carried
out with multiple repositories without running into circular dependency
issues.
Let's say that there's the pulsar repository and the pulsar-connectors
repository. How do you release these if they depend on each other (circular
dependency at the repository level)? Assuming that a git version tag
applies to all files in the complete repository and that the source code
download has to match the repository content, I don't see how a release
could happen if there are circular dependencies at the repository level.
I guess that the new repository https://github.com/apache/pulsar-release
could be part of the solution. Is this documented already somewhere? What
is the role of "pulsar-release" repository?
Another concern I have had is about CI for the multiple repositories. There
are dependencies between the repositories. If an upstream library changes,
do we intend to run the tests in the downstream components to ensure that
the interfaces don't break?
Is the documentation for PIP-62 up-to-date? (PIP-62 link:
https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
)
Please let me know how I can help with the solution. What are the next
steps? Perhaps we could share the tasks and make faster progress together?
BR, Lari
On Mon, Feb 1, 2021 at 11:17 PM Sijie Guo <gu...@gmail.com> wrote:
> Yes. I was doing the work there and hopefully will get there soon.
>
> - Sijie
>
> On Mon, Feb 1, 2021 at 12:40 PM Ali Ahmed <ah...@gmail.com> wrote:
>
> > I recommend we move the connectors away for the pulsar repo to reduce the
> > load on the main ci pipeline. The new repo seems ready.
> > https://github.com/apache/pulsar-connectors.
> >
> > -Ali
> >
> > On Fri, Jan 29, 2021 at 9:22 AM Sijie Guo <gu...@gmail.com> wrote:
> >
> > > Currently, Github Actions are shared across one large `apache`
> > > organization. It is the main problem for GA-based CI besides flaky
> tests.
> > >
> > > If we use Azure Pipeline, we can have a dedicated project for the
> pulsar.
> > > So we will have more resources to run.
> > > It will solve the problem that this proposal tries to solve. The
> approach
> > > has been used by Flink. We have started some experiments. We will share
> > > some of results here next week.
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Fri, Jan 29, 2021 at 8:34 AM Lari Hotari <La...@hotari.net> wrote:
> > >
> > > > Hi Sijie,
> > > >
> > > > Let's keep this work going since resolving the problems with Pulsar
> CI
> > > are
> > > > urgent.
> > > >
> > > > I took a quick glance on the Azure Pipelines solution in Flink. By
> > > Googling
> > > > I found
> > > https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines
> > > > .
> > > > In the repository I found
> > > > https://github.com/apache/flink/blob/master/azure-pipelines.yml
> which
> > > > references
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/jobs-template.yml
> > > >
> > > > It uses the build matrix feature to parallelize the execution:
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/dd0ee24e55dab4ae76201103c76495bc4fa0f73b/tools/azure-pipelines/jobs-template.yml#L88-L107
> > > >
> > > > What would be the key benefit for Pulsar CI of using Azure Pipelines
> > over
> > > > GitHub Actions?
> > > >
> > > > -Lari
> > > >
> > > > On Fri, Jan 29, 2021 at 6:03 PM Sijie Guo <gu...@gmail.com>
> wrote:
> > > >
> > > > > Lari,
> > > > >
> > > > > Yes, we can keep this proposal open for discussion. That's for
> sure.
> > > > >
> > > > > I just don't have any good solution at this moment with a
> > > > multiple-workflow
> > > > > approach using Github Actions.
> > > > >
> > > > > An alternative is to look into Azure Pipeline, which the Flink
> > > community
> > > > is
> > > > > using.
> > > > > We are still learning there. Will post thoughts here once we have a
> > > > better
> > > > > idea.
> > > > >
> > > > > Thanks,
> > > > > Sijie
> > > > >
> > > > > On Fri, Jan 29, 2021 at 5:07 AM Lari Hotari <La...@hotari.net>
> wrote:
> > > > >
> > > > > > Thanks for the feedback, Sijie.
> > > > > >
> > > > > > > If this proposal is blocked by the other proposal, we should
> > focus
> > > on
> > > > > > getting the changes for the other proposal before talking about
> > > merging
> > > > > > them.
> > > > > >
> > > > > > Yes, the current proposal depends on the draft PIP for "Changes
> to
> > > > flaky
> > > > > > test handling". I'll follow up on fixing the flaky test in a new
> > > email
> > > > > > thread.
> > > > > >
> > > > > > I hope we could get the discussions going on both draft PIPs and
> > find
> > > > > > consensus together as a community.
> > > > > > During the discussions, more solution options will come up. Each
> > > > solution
> > > > > > has trade offs.
> > > > > > It would be useful to document the options when the community
> > doesn't
> > > > > > immediately agree on a single choice.
> > > > > > I was thinking that these options could be documented in the same
> > > draft
> > > > > PIP
> > > > > > documents.
> > > > > >
> > > > > > I can give multiple authors editing access to the Google Docs so
> > that
> > > > we
> > > > > > can keep on editing a single document for both draft PIPs.
> > > > > > Anyone who would want to add more solution options to the
> > documents,
> > > > > please
> > > > > > let me know so that I'll add editing access.
> > > > > >
> > > > > > Sijie, would you like to document the option around keeping the
> > > > workflow
> > > > > as
> > > > > > multiple smaller workflows?
> > > > > > I have understood that the problems that have come up with the
> > Pulsar
> > > > CI
> > > > > > regarding resource consumption would have to be resolved in that
> > > > > > alternative as well.
> > > > > >
> > > > > > I believe that everyone is open to any set of solution
> alternatives
> > > > which
> > > > > > solves the problems that we have with Pulsar CI.
> > > > > > We all know that it's urgent to fix Pulsar CI asap. We can do it
> > > > > together.
> > > > > >
> > > > > > BR, Lari
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <gu...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Lari,
> > > > > > >
> > > > > > > Thank you for bringing this proposal up! This is a great
> > > initiative.
> > > > > > >
> > > > > > > However, I agreed with Yong. We have spent tons of effort
> > splitting
> > > > one
> > > > > > > large workflow into multiple smaller workflows.
> > > > > > >
> > > > > > > If this proposal is blocked by the other proposal, we should
> > focus
> > > on
> > > > > > > getting the changes for the other proposal before talking about
> > > > merging
> > > > > > > them.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sijie
> > > > > > >
> > > > > > > On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net>
> > > wrote:
> > > > > > >
> > > > > > > > Thank you for the comments Penghui.
> > > > > > > >
> > > > > > > > Exactly what you said, we should make the tests stable.
> > > > > > > > The proposals in the other draft PIP "Changes to flaky test
> > > > handling"
> > > > > > > deals
> > > > > > > > with that.
> > > > > > > > It's currently a draft and needs more eyes. Would you be able
> > to
> > > > > take a
> > > > > > > > closer look at that too?
> > > > > > > >
> > > > > > > > BR, Lari
> > > > > > > >
> > > > > > > > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <
> > > > codelipenghui@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Currently, especially for the integration tests, a lot time
> > to
> > > > > build
> > > > > > > > > pulsar distributions and docker images.
> > > > > > > > > I think before merge tests we should to make the test
> stable,
> > > > > > otherwise
> > > > > > > > > rerun the test will become more expensive.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Penghui
> > > > > > > > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> > > > > > > zhangyong1025.zy@gmail.com
> > > > > > > > >,
> > > > > > > > > wrote:
> > > > > > > > > > I am not sure that merge all the workflows into one
> > workflow
> > > > is a
> > > > > > > good
> > > > > > > > > > idea. As
> > > > > > > > > > I know, Github Actions doesn't allow to rerun a single
> job
> > > in a
> > > > > > > > workflow.
> > > > > > > > > > That means
> > > > > > > > > > if there has any failure in the workflow, we need to
> rerun
> > > all
> > > > > > > > > > steps/stage. There has
> > > > > > > > > > a worst-case is we failed in the different tests when
> > > rerunning
> > > > > it
> > > > > > > and
> > > > > > > > > this
> > > > > > > > > > would take
> > > > > > > > > > more time to pass the CI.
> > > > > > > > > >
> > > > > > > > > > ---
> > > > > > > > > > Yong
> > > > > > > > > >
> > > > > > > > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <
> > > > lari.hotari@sagire.fi
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > >
> > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > consuming
> > > > > the
> > > > > > > > > majority
> > > > > > > > > > > of the shared pool of resources allocated for
> > > > > github.com/apache
> > > > > > > > > projects.
> > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > demand
> > > > > to
> > > > > > > > > improve
> > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > <
> > > > > >
> https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > >
> > > > > > > > > asap.
> > > > > > > > > > >
> > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > time
> > > > > > that a
> > > > > > > > > Runner
> > > > > > > > > > > is occupied. I observed the workflow runs for handling
> a
> > > > single
> > > > > > > Pull
> > > > > > > > > > > Request (in my personal fork) and these were the
> running
> > > > > > durations:
> > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > >
> > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > Actions
> > > > > > > > > Runners is
> > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> request
> > to
> > > > > > > > > apache/pulsar.*
> > > > > > > > > > >
> > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> is
> > > very
> > > > > > high,
> > > > > > > > > this
> > > > > > > > > > > leads to the build queue to grow and take a long time
> to
> > > > > process.
> > > > > > > > > > >
> > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> for
> > > the
> > > > > > last
> > > > > > > 3
> > > > > > > > > > > months. During this period I worked on a few
> experiments.
> > > The
> > > > > > > > learnings
> > > > > > > > > > > from the past experiments are documented at a high
> level
> > in
> > > > the
> > > > > > > > > following
> > > > > > > > > > > draft PIP document.
> > > > > > > > > > >
> > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > CI"
> > > > > > document
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > > Google doc:*
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > >
> > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > based
> > > > on
> > > > > > the
> > > > > > > > > feedback
> > > > > > > > > > > asap. If there's already a similar effort ongoing, I
> hope
> > > we
> > > > > can
> > > > > > > join
> > > > > > > > > > > efforts.
> > > > > > > > > > >
> > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > >
> > > > > > > > > > > BR, Lari
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > -Ali
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Thank you, Ali. This sounds good!
BR, Lari
On Mon, Feb 1, 2021 at 11:47 PM Ali Ahmed <ah...@gmail.com> wrote:
> We will need some simple connectors without dependencies, to replace the
> existing ones for basic integration testing. I can write those.
>
>
> On Mon, Feb 1, 2021 at 1:17 PM Sijie Guo <gu...@gmail.com> wrote:
>
> > Yes. I was doing the work there and hopefully will get there soon.
> >
> > - Sijie
> >
> > On Mon, Feb 1, 2021 at 12:40 PM Ali Ahmed <ah...@gmail.com> wrote:
> >
> > > I recommend we move the connectors away for the pulsar repo to reduce
> the
> > > load on the main ci pipeline. The new repo seems ready.
> > > https://github.com/apache/pulsar-connectors.
> > >
> > > -Ali
> > >
> > > On Fri, Jan 29, 2021 at 9:22 AM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > > Currently, Github Actions are shared across one large `apache`
> > > > organization. It is the main problem for GA-based CI besides flaky
> > tests.
> > > >
> > > > If we use Azure Pipeline, we can have a dedicated project for the
> > pulsar.
> > > > So we will have more resources to run.
> > > > It will solve the problem that this proposal tries to solve. The
> > approach
> > > > has been used by Flink. We have started some experiments. We will
> share
> > > > some of results here next week.
> > > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Fri, Jan 29, 2021 at 8:34 AM Lari Hotari <La...@hotari.net> wrote:
> > > >
> > > > > Hi Sijie,
> > > > >
> > > > > Let's keep this work going since resolving the problems with Pulsar
> > CI
> > > > are
> > > > > urgent.
> > > > >
> > > > > I took a quick glance on the Azure Pipelines solution in Flink. By
> > > > Googling
> > > > > I found
> > > > https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines
> > > > > .
> > > > > In the repository I found
> > > > > https://github.com/apache/flink/blob/master/azure-pipelines.yml
> > which
> > > > > references
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/jobs-template.yml
> > > > >
> > > > > It uses the build matrix feature to parallelize the execution:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/dd0ee24e55dab4ae76201103c76495bc4fa0f73b/tools/azure-pipelines/jobs-template.yml#L88-L107
> > > > >
> > > > > What would be the key benefit for Pulsar CI of using Azure
> Pipelines
> > > over
> > > > > GitHub Actions?
> > > > >
> > > > > -Lari
> > > > >
> > > > > On Fri, Jan 29, 2021 at 6:03 PM Sijie Guo <gu...@gmail.com>
> > wrote:
> > > > >
> > > > > > Lari,
> > > > > >
> > > > > > Yes, we can keep this proposal open for discussion. That's for
> > sure.
> > > > > >
> > > > > > I just don't have any good solution at this moment with a
> > > > > multiple-workflow
> > > > > > approach using Github Actions.
> > > > > >
> > > > > > An alternative is to look into Azure Pipeline, which the Flink
> > > > community
> > > > > is
> > > > > > using.
> > > > > > We are still learning there. Will post thoughts here once we
> have a
> > > > > better
> > > > > > idea.
> > > > > >
> > > > > > Thanks,
> > > > > > Sijie
> > > > > >
> > > > > > On Fri, Jan 29, 2021 at 5:07 AM Lari Hotari <La...@hotari.net>
> > wrote:
> > > > > >
> > > > > > > Thanks for the feedback, Sijie.
> > > > > > >
> > > > > > > > If this proposal is blocked by the other proposal, we should
> > > focus
> > > > on
> > > > > > > getting the changes for the other proposal before talking about
> > > > merging
> > > > > > > them.
> > > > > > >
> > > > > > > Yes, the current proposal depends on the draft PIP for "Changes
> > to
> > > > > flaky
> > > > > > > test handling". I'll follow up on fixing the flaky test in a
> new
> > > > email
> > > > > > > thread.
> > > > > > >
> > > > > > > I hope we could get the discussions going on both draft PIPs
> and
> > > find
> > > > > > > consensus together as a community.
> > > > > > > During the discussions, more solution options will come up.
> Each
> > > > > solution
> > > > > > > has trade offs.
> > > > > > > It would be useful to document the options when the community
> > > doesn't
> > > > > > > immediately agree on a single choice.
> > > > > > > I was thinking that these options could be documented in the
> same
> > > > draft
> > > > > > PIP
> > > > > > > documents.
> > > > > > >
> > > > > > > I can give multiple authors editing access to the Google Docs
> so
> > > that
> > > > > we
> > > > > > > can keep on editing a single document for both draft PIPs.
> > > > > > > Anyone who would want to add more solution options to the
> > > documents,
> > > > > > please
> > > > > > > let me know so that I'll add editing access.
> > > > > > >
> > > > > > > Sijie, would you like to document the option around keeping the
> > > > > workflow
> > > > > > as
> > > > > > > multiple smaller workflows?
> > > > > > > I have understood that the problems that have come up with the
> > > Pulsar
> > > > > CI
> > > > > > > regarding resource consumption would have to be resolved in
> that
> > > > > > > alternative as well.
> > > > > > >
> > > > > > > I believe that everyone is open to any set of solution
> > alternatives
> > > > > which
> > > > > > > solves the problems that we have with Pulsar CI.
> > > > > > > We all know that it's urgent to fix Pulsar CI asap. We can do
> it
> > > > > > together.
> > > > > > >
> > > > > > > BR, Lari
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <guosijie@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Lari,
> > > > > > > >
> > > > > > > > Thank you for bringing this proposal up! This is a great
> > > > initiative.
> > > > > > > >
> > > > > > > > However, I agreed with Yong. We have spent tons of effort
> > > splitting
> > > > > one
> > > > > > > > large workflow into multiple smaller workflows.
> > > > > > > >
> > > > > > > > If this proposal is blocked by the other proposal, we should
> > > focus
> > > > on
> > > > > > > > getting the changes for the other proposal before talking
> about
> > > > > merging
> > > > > > > > them.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Sijie
> > > > > > > >
> > > > > > > > On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <Lari@hotari.net
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Thank you for the comments Penghui.
> > > > > > > > >
> > > > > > > > > Exactly what you said, we should make the tests stable.
> > > > > > > > > The proposals in the other draft PIP "Changes to flaky test
> > > > > handling"
> > > > > > > > deals
> > > > > > > > > with that.
> > > > > > > > > It's currently a draft and needs more eyes. Would you be
> able
> > > to
> > > > > > take a
> > > > > > > > > closer look at that too?
> > > > > > > > >
> > > > > > > > > BR, Lari
> > > > > > > > >
> > > > > > > > > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <
> > > > > codelipenghui@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Currently, especially for the integration tests, a lot
> time
> > > to
> > > > > > build
> > > > > > > > > > pulsar distributions and docker images.
> > > > > > > > > > I think before merge tests we should to make the test
> > stable,
> > > > > > > otherwise
> > > > > > > > > > rerun the test will become more expensive.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Penghui
> > > > > > > > > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> > > > > > > > zhangyong1025.zy@gmail.com
> > > > > > > > > >,
> > > > > > > > > > wrote:
> > > > > > > > > > > I am not sure that merge all the workflows into one
> > > workflow
> > > > > is a
> > > > > > > > good
> > > > > > > > > > > idea. As
> > > > > > > > > > > I know, Github Actions doesn't allow to rerun a single
> > job
> > > > in a
> > > > > > > > > workflow.
> > > > > > > > > > > That means
> > > > > > > > > > > if there has any failure in the workflow, we need to
> > rerun
> > > > all
> > > > > > > > > > > steps/stage. There has
> > > > > > > > > > > a worst-case is we failed in the different tests when
> > > > rerunning
> > > > > > it
> > > > > > > > and
> > > > > > > > > > this
> > > > > > > > > > > would take
> > > > > > > > > > > more time to pass the CI.
> > > > > > > > > > >
> > > > > > > > > > > ---
> > > > > > > > > > > Yong
> > > > > > > > > > >
> > > > > > > > > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <
> > > > > lari.hotari@sagire.fi
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > > >
> > > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > > consuming
> > > > > > the
> > > > > > > > > > majority
> > > > > > > > > > > > of the shared pool of resources allocated for
> > > > > > github.com/apache
> > > > > > > > > > projects.
> > > > > > > > > > > > Other Apache projects have been impacted and there
> is a
> > > > > demand
> > > > > > to
> > > > > > > > > > improve
> > > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > > <
> > > > > > >
> > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > > >
> > > > > > > > > > asap.
> > > > > > > > > > > >
> > > > > > > > > > > > In GitHub Actions Runners, the unit of resources is
> the
> > > > time
> > > > > > > that a
> > > > > > > > > > Runner
> > > > > > > > > > > > is occupied. I observed the workflow runs for
> handling
> > a
> > > > > single
> > > > > > > > Pull
> > > > > > > > > > > > Request (in my personal fork) and these were the
> > running
> > > > > > > durations:
> > > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > > >
> > > > > > > > > > > > *In this case, the total resource consumption of
> GitHub
> > > > > Actions
> > > > > > > > > > Runners is
> > > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> > request
> > > to
> > > > > > > > > > apache/pulsar.*
> > > > > > > > > > > >
> > > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> > is
> > > > very
> > > > > > > high,
> > > > > > > > > > this
> > > > > > > > > > > > leads to the build queue to grow and take a long time
> > to
> > > > > > process.
> > > > > > > > > > > >
> > > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> > for
> > > > the
> > > > > > > last
> > > > > > > > 3
> > > > > > > > > > > > months. During this period I worked on a few
> > experiments.
> > > > The
> > > > > > > > > learnings
> > > > > > > > > > > > from the past experiments are documented at a high
> > level
> > > in
> > > > > the
> > > > > > > > > > following
> > > > > > > > > > > > draft PIP document.
> > > > > > > > > > > >
> > > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based
> Pulsar
> > > CI"
> > > > > > > document
> > > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > > > Google doc:*
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > > >
> > > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > > based
> > > > > on
> > > > > > > the
> > > > > > > > > > feedback
> > > > > > > > > > > > asap. If there's already a similar effort ongoing, I
> > hope
> > > > we
> > > > > > can
> > > > > > > > join
> > > > > > > > > > > > efforts.
> > > > > > > > > > > >
> > > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > > >
> > > > > > > > > > > > BR, Lari
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > -Ali
> > >
> >
>
>
> --
> -Ali
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Ali Ahmed <ah...@gmail.com>.
We will need some simple connectors without dependencies, to replace the
existing ones for basic integration testing. I can write those.
On Mon, Feb 1, 2021 at 1:17 PM Sijie Guo <gu...@gmail.com> wrote:
> Yes. I was doing the work there and hopefully will get there soon.
>
> - Sijie
>
> On Mon, Feb 1, 2021 at 12:40 PM Ali Ahmed <ah...@gmail.com> wrote:
>
> > I recommend we move the connectors away for the pulsar repo to reduce the
> > load on the main ci pipeline. The new repo seems ready.
> > https://github.com/apache/pulsar-connectors.
> >
> > -Ali
> >
> > On Fri, Jan 29, 2021 at 9:22 AM Sijie Guo <gu...@gmail.com> wrote:
> >
> > > Currently, Github Actions are shared across one large `apache`
> > > organization. It is the main problem for GA-based CI besides flaky
> tests.
> > >
> > > If we use Azure Pipeline, we can have a dedicated project for the
> pulsar.
> > > So we will have more resources to run.
> > > It will solve the problem that this proposal tries to solve. The
> approach
> > > has been used by Flink. We have started some experiments. We will share
> > > some of results here next week.
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Fri, Jan 29, 2021 at 8:34 AM Lari Hotari <La...@hotari.net> wrote:
> > >
> > > > Hi Sijie,
> > > >
> > > > Let's keep this work going since resolving the problems with Pulsar
> CI
> > > are
> > > > urgent.
> > > >
> > > > I took a quick glance on the Azure Pipelines solution in Flink. By
> > > Googling
> > > > I found
> > > https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines
> > > > .
> > > > In the repository I found
> > > > https://github.com/apache/flink/blob/master/azure-pipelines.yml
> which
> > > > references
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/jobs-template.yml
> > > >
> > > > It uses the build matrix feature to parallelize the execution:
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/dd0ee24e55dab4ae76201103c76495bc4fa0f73b/tools/azure-pipelines/jobs-template.yml#L88-L107
> > > >
> > > > What would be the key benefit for Pulsar CI of using Azure Pipelines
> > over
> > > > GitHub Actions?
> > > >
> > > > -Lari
> > > >
> > > > On Fri, Jan 29, 2021 at 6:03 PM Sijie Guo <gu...@gmail.com>
> wrote:
> > > >
> > > > > Lari,
> > > > >
> > > > > Yes, we can keep this proposal open for discussion. That's for
> sure.
> > > > >
> > > > > I just don't have any good solution at this moment with a
> > > > multiple-workflow
> > > > > approach using Github Actions.
> > > > >
> > > > > An alternative is to look into Azure Pipeline, which the Flink
> > > community
> > > > is
> > > > > using.
> > > > > We are still learning there. Will post thoughts here once we have a
> > > > better
> > > > > idea.
> > > > >
> > > > > Thanks,
> > > > > Sijie
> > > > >
> > > > > On Fri, Jan 29, 2021 at 5:07 AM Lari Hotari <La...@hotari.net>
> wrote:
> > > > >
> > > > > > Thanks for the feedback, Sijie.
> > > > > >
> > > > > > > If this proposal is blocked by the other proposal, we should
> > focus
> > > on
> > > > > > getting the changes for the other proposal before talking about
> > > merging
> > > > > > them.
> > > > > >
> > > > > > Yes, the current proposal depends on the draft PIP for "Changes
> to
> > > > flaky
> > > > > > test handling". I'll follow up on fixing the flaky test in a new
> > > email
> > > > > > thread.
> > > > > >
> > > > > > I hope we could get the discussions going on both draft PIPs and
> > find
> > > > > > consensus together as a community.
> > > > > > During the discussions, more solution options will come up. Each
> > > > solution
> > > > > > has trade offs.
> > > > > > It would be useful to document the options when the community
> > doesn't
> > > > > > immediately agree on a single choice.
> > > > > > I was thinking that these options could be documented in the same
> > > draft
> > > > > PIP
> > > > > > documents.
> > > > > >
> > > > > > I can give multiple authors editing access to the Google Docs so
> > that
> > > > we
> > > > > > can keep on editing a single document for both draft PIPs.
> > > > > > Anyone who would want to add more solution options to the
> > documents,
> > > > > please
> > > > > > let me know so that I'll add editing access.
> > > > > >
> > > > > > Sijie, would you like to document the option around keeping the
> > > > workflow
> > > > > as
> > > > > > multiple smaller workflows?
> > > > > > I have understood that the problems that have come up with the
> > Pulsar
> > > > CI
> > > > > > regarding resource consumption would have to be resolved in that
> > > > > > alternative as well.
> > > > > >
> > > > > > I believe that everyone is open to any set of solution
> alternatives
> > > > which
> > > > > > solves the problems that we have with Pulsar CI.
> > > > > > We all know that it's urgent to fix Pulsar CI asap. We can do it
> > > > > together.
> > > > > >
> > > > > > BR, Lari
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <gu...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Lari,
> > > > > > >
> > > > > > > Thank you for bringing this proposal up! This is a great
> > > initiative.
> > > > > > >
> > > > > > > However, I agreed with Yong. We have spent tons of effort
> > splitting
> > > > one
> > > > > > > large workflow into multiple smaller workflows.
> > > > > > >
> > > > > > > If this proposal is blocked by the other proposal, we should
> > focus
> > > on
> > > > > > > getting the changes for the other proposal before talking about
> > > > merging
> > > > > > > them.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sijie
> > > > > > >
> > > > > > > On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net>
> > > wrote:
> > > > > > >
> > > > > > > > Thank you for the comments Penghui.
> > > > > > > >
> > > > > > > > Exactly what you said, we should make the tests stable.
> > > > > > > > The proposals in the other draft PIP "Changes to flaky test
> > > > handling"
> > > > > > > deals
> > > > > > > > with that.
> > > > > > > > It's currently a draft and needs more eyes. Would you be able
> > to
> > > > > take a
> > > > > > > > closer look at that too?
> > > > > > > >
> > > > > > > > BR, Lari
> > > > > > > >
> > > > > > > > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <
> > > > codelipenghui@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Currently, especially for the integration tests, a lot time
> > to
> > > > > build
> > > > > > > > > pulsar distributions and docker images.
> > > > > > > > > I think before merge tests we should to make the test
> stable,
> > > > > > otherwise
> > > > > > > > > rerun the test will become more expensive.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Penghui
> > > > > > > > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> > > > > > > zhangyong1025.zy@gmail.com
> > > > > > > > >,
> > > > > > > > > wrote:
> > > > > > > > > > I am not sure that merge all the workflows into one
> > workflow
> > > > is a
> > > > > > > good
> > > > > > > > > > idea. As
> > > > > > > > > > I know, Github Actions doesn't allow to rerun a single
> job
> > > in a
> > > > > > > > workflow.
> > > > > > > > > > That means
> > > > > > > > > > if there has any failure in the workflow, we need to
> rerun
> > > all
> > > > > > > > > > steps/stage. There has
> > > > > > > > > > a worst-case is we failed in the different tests when
> > > rerunning
> > > > > it
> > > > > > > and
> > > > > > > > > this
> > > > > > > > > > would take
> > > > > > > > > > more time to pass the CI.
> > > > > > > > > >
> > > > > > > > > > ---
> > > > > > > > > > Yong
> > > > > > > > > >
> > > > > > > > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <
> > > > lari.hotari@sagire.fi
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > > >
> > > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > > consuming
> > > > > the
> > > > > > > > > majority
> > > > > > > > > > > of the shared pool of resources allocated for
> > > > > github.com/apache
> > > > > > > > > projects.
> > > > > > > > > > > Other Apache projects have been impacted and there is a
> > > > demand
> > > > > to
> > > > > > > > > improve
> > > > > > > > > > > the Pulsar CI
> > > > > > > > > > > <
> > > > > >
> https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > > >
> > > > > > > > > asap.
> > > > > > > > > > >
> > > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > > time
> > > > > > that a
> > > > > > > > > Runner
> > > > > > > > > > > is occupied. I observed the workflow runs for handling
> a
> > > > single
> > > > > > > Pull
> > > > > > > > > > > Request (in my personal fork) and these were the
> running
> > > > > > durations:
> > > > > > > > > > > Workflow name Duration
> > > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > > >
> > > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > > Actions
> > > > > > > > > Runners is
> > > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull
> request
> > to
> > > > > > > > > apache/pulsar.*
> > > > > > > > > > >
> > > > > > > > > > > Since GitHub Actions Runner resource pool utilization
> is
> > > very
> > > > > > high,
> > > > > > > > > this
> > > > > > > > > > > leads to the build queue to grow and take a long time
> to
> > > > > process.
> > > > > > > > > > >
> > > > > > > > > > > I have been looking for ways to improve the Pulsar CI
> for
> > > the
> > > > > > last
> > > > > > > 3
> > > > > > > > > > > months. During this period I worked on a few
> experiments.
> > > The
> > > > > > > > learnings
> > > > > > > > > > > from the past experiments are documented at a high
> level
> > in
> > > > the
> > > > > > > > > following
> > > > > > > > > > > draft PIP document.
> > > > > > > > > > >
> > > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> > CI"
> > > > > > document
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > > Google doc:*
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > > >
> > > > > > > > > > > *Please participate* so that we get the plan adjusted
> > based
> > > > on
> > > > > > the
> > > > > > > > > feedback
> > > > > > > > > > > asap. If there's already a similar effort ongoing, I
> hope
> > > we
> > > > > can
> > > > > > > join
> > > > > > > > > > > efforts.
> > > > > > > > > > >
> > > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > > >
> > > > > > > > > > > BR, Lari
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > -Ali
> >
>
--
-Ali
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Sijie Guo <gu...@gmail.com>.
Yes. I was doing the work there and hopefully will get there soon.
- Sijie
On Mon, Feb 1, 2021 at 12:40 PM Ali Ahmed <ah...@gmail.com> wrote:
> I recommend we move the connectors away for the pulsar repo to reduce the
> load on the main ci pipeline. The new repo seems ready.
> https://github.com/apache/pulsar-connectors.
>
> -Ali
>
> On Fri, Jan 29, 2021 at 9:22 AM Sijie Guo <gu...@gmail.com> wrote:
>
> > Currently, Github Actions are shared across one large `apache`
> > organization. It is the main problem for GA-based CI besides flaky tests.
> >
> > If we use Azure Pipeline, we can have a dedicated project for the pulsar.
> > So we will have more resources to run.
> > It will solve the problem that this proposal tries to solve. The approach
> > has been used by Flink. We have started some experiments. We will share
> > some of results here next week.
> >
> > Thanks,
> > Sijie
> >
> > On Fri, Jan 29, 2021 at 8:34 AM Lari Hotari <La...@hotari.net> wrote:
> >
> > > Hi Sijie,
> > >
> > > Let's keep this work going since resolving the problems with Pulsar CI
> > are
> > > urgent.
> > >
> > > I took a quick glance on the Azure Pipelines solution in Flink. By
> > Googling
> > > I found
> > https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines
> > > .
> > > In the repository I found
> > > https://github.com/apache/flink/blob/master/azure-pipelines.yml which
> > > references
> > >
> > >
> >
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/jobs-template.yml
> > >
> > > It uses the build matrix feature to parallelize the execution:
> > >
> > >
> >
> https://github.com/apache/flink/blob/dd0ee24e55dab4ae76201103c76495bc4fa0f73b/tools/azure-pipelines/jobs-template.yml#L88-L107
> > >
> > > What would be the key benefit for Pulsar CI of using Azure Pipelines
> over
> > > GitHub Actions?
> > >
> > > -Lari
> > >
> > > On Fri, Jan 29, 2021 at 6:03 PM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > > Lari,
> > > >
> > > > Yes, we can keep this proposal open for discussion. That's for sure.
> > > >
> > > > I just don't have any good solution at this moment with a
> > > multiple-workflow
> > > > approach using Github Actions.
> > > >
> > > > An alternative is to look into Azure Pipeline, which the Flink
> > community
> > > is
> > > > using.
> > > > We are still learning there. Will post thoughts here once we have a
> > > better
> > > > idea.
> > > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Fri, Jan 29, 2021 at 5:07 AM Lari Hotari <La...@hotari.net> wrote:
> > > >
> > > > > Thanks for the feedback, Sijie.
> > > > >
> > > > > > If this proposal is blocked by the other proposal, we should
> focus
> > on
> > > > > getting the changes for the other proposal before talking about
> > merging
> > > > > them.
> > > > >
> > > > > Yes, the current proposal depends on the draft PIP for "Changes to
> > > flaky
> > > > > test handling". I'll follow up on fixing the flaky test in a new
> > email
> > > > > thread.
> > > > >
> > > > > I hope we could get the discussions going on both draft PIPs and
> find
> > > > > consensus together as a community.
> > > > > During the discussions, more solution options will come up. Each
> > > solution
> > > > > has trade offs.
> > > > > It would be useful to document the options when the community
> doesn't
> > > > > immediately agree on a single choice.
> > > > > I was thinking that these options could be documented in the same
> > draft
> > > > PIP
> > > > > documents.
> > > > >
> > > > > I can give multiple authors editing access to the Google Docs so
> that
> > > we
> > > > > can keep on editing a single document for both draft PIPs.
> > > > > Anyone who would want to add more solution options to the
> documents,
> > > > please
> > > > > let me know so that I'll add editing access.
> > > > >
> > > > > Sijie, would you like to document the option around keeping the
> > > workflow
> > > > as
> > > > > multiple smaller workflows?
> > > > > I have understood that the problems that have come up with the
> Pulsar
> > > CI
> > > > > regarding resource consumption would have to be resolved in that
> > > > > alternative as well.
> > > > >
> > > > > I believe that everyone is open to any set of solution alternatives
> > > which
> > > > > solves the problems that we have with Pulsar CI.
> > > > > We all know that it's urgent to fix Pulsar CI asap. We can do it
> > > > together.
> > > > >
> > > > > BR, Lari
> > > > >
> > > > >
> > > > > On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <gu...@gmail.com>
> > wrote:
> > > > >
> > > > > > Lari,
> > > > > >
> > > > > > Thank you for bringing this proposal up! This is a great
> > initiative.
> > > > > >
> > > > > > However, I agreed with Yong. We have spent tons of effort
> splitting
> > > one
> > > > > > large workflow into multiple smaller workflows.
> > > > > >
> > > > > > If this proposal is blocked by the other proposal, we should
> focus
> > on
> > > > > > getting the changes for the other proposal before talking about
> > > merging
> > > > > > them.
> > > > > >
> > > > > > Thanks,
> > > > > > Sijie
> > > > > >
> > > > > > On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net>
> > wrote:
> > > > > >
> > > > > > > Thank you for the comments Penghui.
> > > > > > >
> > > > > > > Exactly what you said, we should make the tests stable.
> > > > > > > The proposals in the other draft PIP "Changes to flaky test
> > > handling"
> > > > > > deals
> > > > > > > with that.
> > > > > > > It's currently a draft and needs more eyes. Would you be able
> to
> > > > take a
> > > > > > > closer look at that too?
> > > > > > >
> > > > > > > BR, Lari
> > > > > > >
> > > > > > > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <
> > > codelipenghui@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Currently, especially for the integration tests, a lot time
> to
> > > > build
> > > > > > > > pulsar distributions and docker images.
> > > > > > > > I think before merge tests we should to make the test stable,
> > > > > otherwise
> > > > > > > > rerun the test will become more expensive.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Penghui
> > > > > > > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> > > > > > zhangyong1025.zy@gmail.com
> > > > > > > >,
> > > > > > > > wrote:
> > > > > > > > > I am not sure that merge all the workflows into one
> workflow
> > > is a
> > > > > > good
> > > > > > > > > idea. As
> > > > > > > > > I know, Github Actions doesn't allow to rerun a single job
> > in a
> > > > > > > workflow.
> > > > > > > > > That means
> > > > > > > > > if there has any failure in the workflow, we need to rerun
> > all
> > > > > > > > > steps/stage. There has
> > > > > > > > > a worst-case is we failed in the different tests when
> > rerunning
> > > > it
> > > > > > and
> > > > > > > > this
> > > > > > > > > would take
> > > > > > > > > more time to pass the CI.
> > > > > > > > >
> > > > > > > > > ---
> > > > > > > > > Yong
> > > > > > > > >
> > > > > > > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <
> > > lari.hotari@sagire.fi
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Dear Pulsar community members,
> > > > > > > > > >
> > > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> > consuming
> > > > the
> > > > > > > > majority
> > > > > > > > > > of the shared pool of resources allocated for
> > > > github.com/apache
> > > > > > > > projects.
> > > > > > > > > > Other Apache projects have been impacted and there is a
> > > demand
> > > > to
> > > > > > > > improve
> > > > > > > > > > the Pulsar CI
> > > > > > > > > > <
> > > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > > >
> > > > > > > > asap.
> > > > > > > > > >
> > > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> > time
> > > > > that a
> > > > > > > > Runner
> > > > > > > > > > is occupied. I observed the workflow runs for handling a
> > > single
> > > > > > Pull
> > > > > > > > > > Request (in my personal fork) and these were the running
> > > > > durations:
> > > > > > > > > > Workflow name Duration
> > > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > > TOTAL 19:36:50
> > > > > > > > > >
> > > > > > > > > > *In this case, the total resource consumption of GitHub
> > > Actions
> > > > > > > > Runners is
> > > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull request
> to
> > > > > > > > apache/pulsar.*
> > > > > > > > > >
> > > > > > > > > > Since GitHub Actions Runner resource pool utilization is
> > very
> > > > > high,
> > > > > > > > this
> > > > > > > > > > leads to the build queue to grow and take a long time to
> > > > process.
> > > > > > > > > >
> > > > > > > > > > I have been looking for ways to improve the Pulsar CI for
> > the
> > > > > last
> > > > > > 3
> > > > > > > > > > months. During this period I worked on a few experiments.
> > The
> > > > > > > learnings
> > > > > > > > > > from the past experiments are documented at a high level
> in
> > > the
> > > > > > > > following
> > > > > > > > > > draft PIP document.
> > > > > > > > > >
> > > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar
> CI"
> > > > > document
> > > > > > > is
> > > > > > > > a
> > > > > > > > > > Google doc:*
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > > >
> > > > > > > > > > *Please participate* so that we get the plan adjusted
> based
> > > on
> > > > > the
> > > > > > > > feedback
> > > > > > > > > > asap. If there's already a similar effort ongoing, I hope
> > we
> > > > can
> > > > > > join
> > > > > > > > > > efforts.
> > > > > > > > > >
> > > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > > >
> > > > > > > > > > BR, Lari
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> -Ali
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Ali Ahmed <ah...@gmail.com>.
I recommend we move the connectors away for the pulsar repo to reduce the
load on the main ci pipeline. The new repo seems ready.
https://github.com/apache/pulsar-connectors.
-Ali
On Fri, Jan 29, 2021 at 9:22 AM Sijie Guo <gu...@gmail.com> wrote:
> Currently, Github Actions are shared across one large `apache`
> organization. It is the main problem for GA-based CI besides flaky tests.
>
> If we use Azure Pipeline, we can have a dedicated project for the pulsar.
> So we will have more resources to run.
> It will solve the problem that this proposal tries to solve. The approach
> has been used by Flink. We have started some experiments. We will share
> some of results here next week.
>
> Thanks,
> Sijie
>
> On Fri, Jan 29, 2021 at 8:34 AM Lari Hotari <La...@hotari.net> wrote:
>
> > Hi Sijie,
> >
> > Let's keep this work going since resolving the problems with Pulsar CI
> are
> > urgent.
> >
> > I took a quick glance on the Azure Pipelines solution in Flink. By
> Googling
> > I found
> https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines
> > .
> > In the repository I found
> > https://github.com/apache/flink/blob/master/azure-pipelines.yml which
> > references
> >
> >
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/jobs-template.yml
> >
> > It uses the build matrix feature to parallelize the execution:
> >
> >
> https://github.com/apache/flink/blob/dd0ee24e55dab4ae76201103c76495bc4fa0f73b/tools/azure-pipelines/jobs-template.yml#L88-L107
> >
> > What would be the key benefit for Pulsar CI of using Azure Pipelines over
> > GitHub Actions?
> >
> > -Lari
> >
> > On Fri, Jan 29, 2021 at 6:03 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > > Lari,
> > >
> > > Yes, we can keep this proposal open for discussion. That's for sure.
> > >
> > > I just don't have any good solution at this moment with a
> > multiple-workflow
> > > approach using Github Actions.
> > >
> > > An alternative is to look into Azure Pipeline, which the Flink
> community
> > is
> > > using.
> > > We are still learning there. Will post thoughts here once we have a
> > better
> > > idea.
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Fri, Jan 29, 2021 at 5:07 AM Lari Hotari <La...@hotari.net> wrote:
> > >
> > > > Thanks for the feedback, Sijie.
> > > >
> > > > > If this proposal is blocked by the other proposal, we should focus
> on
> > > > getting the changes for the other proposal before talking about
> merging
> > > > them.
> > > >
> > > > Yes, the current proposal depends on the draft PIP for "Changes to
> > flaky
> > > > test handling". I'll follow up on fixing the flaky test in a new
> email
> > > > thread.
> > > >
> > > > I hope we could get the discussions going on both draft PIPs and find
> > > > consensus together as a community.
> > > > During the discussions, more solution options will come up. Each
> > solution
> > > > has trade offs.
> > > > It would be useful to document the options when the community doesn't
> > > > immediately agree on a single choice.
> > > > I was thinking that these options could be documented in the same
> draft
> > > PIP
> > > > documents.
> > > >
> > > > I can give multiple authors editing access to the Google Docs so that
> > we
> > > > can keep on editing a single document for both draft PIPs.
> > > > Anyone who would want to add more solution options to the documents,
> > > please
> > > > let me know so that I'll add editing access.
> > > >
> > > > Sijie, would you like to document the option around keeping the
> > workflow
> > > as
> > > > multiple smaller workflows?
> > > > I have understood that the problems that have come up with the Pulsar
> > CI
> > > > regarding resource consumption would have to be resolved in that
> > > > alternative as well.
> > > >
> > > > I believe that everyone is open to any set of solution alternatives
> > which
> > > > solves the problems that we have with Pulsar CI.
> > > > We all know that it's urgent to fix Pulsar CI asap. We can do it
> > > together.
> > > >
> > > > BR, Lari
> > > >
> > > >
> > > > On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <gu...@gmail.com>
> wrote:
> > > >
> > > > > Lari,
> > > > >
> > > > > Thank you for bringing this proposal up! This is a great
> initiative.
> > > > >
> > > > > However, I agreed with Yong. We have spent tons of effort splitting
> > one
> > > > > large workflow into multiple smaller workflows.
> > > > >
> > > > > If this proposal is blocked by the other proposal, we should focus
> on
> > > > > getting the changes for the other proposal before talking about
> > merging
> > > > > them.
> > > > >
> > > > > Thanks,
> > > > > Sijie
> > > > >
> > > > > On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net>
> wrote:
> > > > >
> > > > > > Thank you for the comments Penghui.
> > > > > >
> > > > > > Exactly what you said, we should make the tests stable.
> > > > > > The proposals in the other draft PIP "Changes to flaky test
> > handling"
> > > > > deals
> > > > > > with that.
> > > > > > It's currently a draft and needs more eyes. Would you be able to
> > > take a
> > > > > > closer look at that too?
> > > > > >
> > > > > > BR, Lari
> > > > > >
> > > > > > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <
> > codelipenghui@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Currently, especially for the integration tests, a lot time to
> > > build
> > > > > > > pulsar distributions and docker images.
> > > > > > > I think before merge tests we should to make the test stable,
> > > > otherwise
> > > > > > > rerun the test will become more expensive.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Penghui
> > > > > > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> > > > > zhangyong1025.zy@gmail.com
> > > > > > >,
> > > > > > > wrote:
> > > > > > > > I am not sure that merge all the workflows into one workflow
> > is a
> > > > > good
> > > > > > > > idea. As
> > > > > > > > I know, Github Actions doesn't allow to rerun a single job
> in a
> > > > > > workflow.
> > > > > > > > That means
> > > > > > > > if there has any failure in the workflow, we need to rerun
> all
> > > > > > > > steps/stage. There has
> > > > > > > > a worst-case is we failed in the different tests when
> rerunning
> > > it
> > > > > and
> > > > > > > this
> > > > > > > > would take
> > > > > > > > more time to pass the CI.
> > > > > > > >
> > > > > > > > ---
> > > > > > > > Yong
> > > > > > > >
> > > > > > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <
> > lari.hotari@sagire.fi
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Dear Pulsar community members,
> > > > > > > > >
> > > > > > > > > Currently, the Pulsar GitHub Actions workflows are
> consuming
> > > the
> > > > > > > majority
> > > > > > > > > of the shared pool of resources allocated for
> > > github.com/apache
> > > > > > > projects.
> > > > > > > > > Other Apache projects have been impacted and there is a
> > demand
> > > to
> > > > > > > improve
> > > > > > > > > the Pulsar CI
> > > > > > > > > <
> > > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > > >
> > > > > > > asap.
> > > > > > > > >
> > > > > > > > > In GitHub Actions Runners, the unit of resources is the
> time
> > > > that a
> > > > > > > Runner
> > > > > > > > > is occupied. I observed the workflow runs for handling a
> > single
> > > > > Pull
> > > > > > > > > Request (in my personal fork) and these were the running
> > > > durations:
> > > > > > > > > Workflow name Duration
> > > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > > CI - Misc 0:16:51
> > > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > > CI - Unit 0:42:11
> > > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > > TOTAL 19:36:50
> > > > > > > > >
> > > > > > > > > *In this case, the total resource consumption of GitHub
> > Actions
> > > > > > > Runners is
> > > > > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > > > > apache/pulsar.*
> > > > > > > > >
> > > > > > > > > Since GitHub Actions Runner resource pool utilization is
> very
> > > > high,
> > > > > > > this
> > > > > > > > > leads to the build queue to grow and take a long time to
> > > process.
> > > > > > > > >
> > > > > > > > > I have been looking for ways to improve the Pulsar CI for
> the
> > > > last
> > > > > 3
> > > > > > > > > months. During this period I worked on a few experiments.
> The
> > > > > > learnings
> > > > > > > > > from the past experiments are documented at a high level in
> > the
> > > > > > > following
> > > > > > > > > draft PIP document.
> > > > > > > > >
> > > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> > > > document
> > > > > > is
> > > > > > > a
> > > > > > > > > Google doc:*
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > > >
> > > > > > > > > *Please participate* so that we get the plan adjusted based
> > on
> > > > the
> > > > > > > feedback
> > > > > > > > > asap. If there's already a similar effort ongoing, I hope
> we
> > > can
> > > > > join
> > > > > > > > > efforts.
> > > > > > > > >
> > > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > > >
> > > > > > > > > BR, Lari
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
-Ali
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Sijie Guo <gu...@gmail.com>.
Currently, Github Actions are shared across one large `apache`
organization. It is the main problem for GA-based CI besides flaky tests.
If we use Azure Pipeline, we can have a dedicated project for the pulsar.
So we will have more resources to run.
It will solve the problem that this proposal tries to solve. The approach
has been used by Flink. We have started some experiments. We will share
some of results here next week.
Thanks,
Sijie
On Fri, Jan 29, 2021 at 8:34 AM Lari Hotari <La...@hotari.net> wrote:
> Hi Sijie,
>
> Let's keep this work going since resolving the problems with Pulsar CI are
> urgent.
>
> I took a quick glance on the Azure Pipelines solution in Flink. By Googling
> I found https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines
> .
> In the repository I found
> https://github.com/apache/flink/blob/master/azure-pipelines.yml which
> references
>
> https://github.com/apache/flink/blob/master/tools/azure-pipelines/jobs-template.yml
>
> It uses the build matrix feature to parallelize the execution:
>
> https://github.com/apache/flink/blob/dd0ee24e55dab4ae76201103c76495bc4fa0f73b/tools/azure-pipelines/jobs-template.yml#L88-L107
>
> What would be the key benefit for Pulsar CI of using Azure Pipelines over
> GitHub Actions?
>
> -Lari
>
> On Fri, Jan 29, 2021 at 6:03 PM Sijie Guo <gu...@gmail.com> wrote:
>
> > Lari,
> >
> > Yes, we can keep this proposal open for discussion. That's for sure.
> >
> > I just don't have any good solution at this moment with a
> multiple-workflow
> > approach using Github Actions.
> >
> > An alternative is to look into Azure Pipeline, which the Flink community
> is
> > using.
> > We are still learning there. Will post thoughts here once we have a
> better
> > idea.
> >
> > Thanks,
> > Sijie
> >
> > On Fri, Jan 29, 2021 at 5:07 AM Lari Hotari <La...@hotari.net> wrote:
> >
> > > Thanks for the feedback, Sijie.
> > >
> > > > If this proposal is blocked by the other proposal, we should focus on
> > > getting the changes for the other proposal before talking about merging
> > > them.
> > >
> > > Yes, the current proposal depends on the draft PIP for "Changes to
> flaky
> > > test handling". I'll follow up on fixing the flaky test in a new email
> > > thread.
> > >
> > > I hope we could get the discussions going on both draft PIPs and find
> > > consensus together as a community.
> > > During the discussions, more solution options will come up. Each
> solution
> > > has trade offs.
> > > It would be useful to document the options when the community doesn't
> > > immediately agree on a single choice.
> > > I was thinking that these options could be documented in the same draft
> > PIP
> > > documents.
> > >
> > > I can give multiple authors editing access to the Google Docs so that
> we
> > > can keep on editing a single document for both draft PIPs.
> > > Anyone who would want to add more solution options to the documents,
> > please
> > > let me know so that I'll add editing access.
> > >
> > > Sijie, would you like to document the option around keeping the
> workflow
> > as
> > > multiple smaller workflows?
> > > I have understood that the problems that have come up with the Pulsar
> CI
> > > regarding resource consumption would have to be resolved in that
> > > alternative as well.
> > >
> > > I believe that everyone is open to any set of solution alternatives
> which
> > > solves the problems that we have with Pulsar CI.
> > > We all know that it's urgent to fix Pulsar CI asap. We can do it
> > together.
> > >
> > > BR, Lari
> > >
> > >
> > > On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > > Lari,
> > > >
> > > > Thank you for bringing this proposal up! This is a great initiative.
> > > >
> > > > However, I agreed with Yong. We have spent tons of effort splitting
> one
> > > > large workflow into multiple smaller workflows.
> > > >
> > > > If this proposal is blocked by the other proposal, we should focus on
> > > > getting the changes for the other proposal before talking about
> merging
> > > > them.
> > > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net> wrote:
> > > >
> > > > > Thank you for the comments Penghui.
> > > > >
> > > > > Exactly what you said, we should make the tests stable.
> > > > > The proposals in the other draft PIP "Changes to flaky test
> handling"
> > > > deals
> > > > > with that.
> > > > > It's currently a draft and needs more eyes. Would you be able to
> > take a
> > > > > closer look at that too?
> > > > >
> > > > > BR, Lari
> > > > >
> > > > > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <
> codelipenghui@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Currently, especially for the integration tests, a lot time to
> > build
> > > > > > pulsar distributions and docker images.
> > > > > > I think before merge tests we should to make the test stable,
> > > otherwise
> > > > > > rerun the test will become more expensive.
> > > > > >
> > > > > > Thanks,
> > > > > > Penghui
> > > > > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> > > > zhangyong1025.zy@gmail.com
> > > > > >,
> > > > > > wrote:
> > > > > > > I am not sure that merge all the workflows into one workflow
> is a
> > > > good
> > > > > > > idea. As
> > > > > > > I know, Github Actions doesn't allow to rerun a single job in a
> > > > > workflow.
> > > > > > > That means
> > > > > > > if there has any failure in the workflow, we need to rerun all
> > > > > > > steps/stage. There has
> > > > > > > a worst-case is we failed in the different tests when rerunning
> > it
> > > > and
> > > > > > this
> > > > > > > would take
> > > > > > > more time to pass the CI.
> > > > > > >
> > > > > > > ---
> > > > > > > Yong
> > > > > > >
> > > > > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <
> lari.hotari@sagire.fi
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Dear Pulsar community members,
> > > > > > > >
> > > > > > > > Currently, the Pulsar GitHub Actions workflows are consuming
> > the
> > > > > > majority
> > > > > > > > of the shared pool of resources allocated for
> > github.com/apache
> > > > > > projects.
> > > > > > > > Other Apache projects have been impacted and there is a
> demand
> > to
> > > > > > improve
> > > > > > > > the Pulsar CI
> > > > > > > > <
> > > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > > >
> > > > > > asap.
> > > > > > > >
> > > > > > > > In GitHub Actions Runners, the unit of resources is the time
> > > that a
> > > > > > Runner
> > > > > > > > is occupied. I observed the workflow runs for handling a
> single
> > > > Pull
> > > > > > > > Request (in my personal fork) and these were the running
> > > durations:
> > > > > > > > Workflow name Duration
> > > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > > CI - Misc 0:16:51
> > > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > > CI - Unit 0:42:11
> > > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > > CI - Integration - Process 1:11:23
> > > > > > > > CI - Shade - Test 1:15:45
> > > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > > TOTAL 19:36:50
> > > > > > > >
> > > > > > > > *In this case, the total resource consumption of GitHub
> Actions
> > > > > > Runners is
> > > > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > > > apache/pulsar.*
> > > > > > > >
> > > > > > > > Since GitHub Actions Runner resource pool utilization is very
> > > high,
> > > > > > this
> > > > > > > > leads to the build queue to grow and take a long time to
> > process.
> > > > > > > >
> > > > > > > > I have been looking for ways to improve the Pulsar CI for the
> > > last
> > > > 3
> > > > > > > > months. During this period I worked on a few experiments. The
> > > > > learnings
> > > > > > > > from the past experiments are documented at a high level in
> the
> > > > > > following
> > > > > > > > draft PIP document.
> > > > > > > >
> > > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> > > document
> > > > > is
> > > > > > a
> > > > > > > > Google doc:*
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > > >
> > > > > > > > *Please participate* so that we get the plan adjusted based
> on
> > > the
> > > > > > feedback
> > > > > > > > asap. If there's already a similar effort ongoing, I hope we
> > can
> > > > join
> > > > > > > > efforts.
> > > > > > > >
> > > > > > > > *Let's fix Pulsar CI!*
> > > > > > > >
> > > > > > > > BR, Lari
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Hi Sijie,
Let's keep this work going since resolving the problems with Pulsar CI are
urgent.
I took a quick glance on the Azure Pipelines solution in Flink. By Googling
I found https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines .
In the repository I found
https://github.com/apache/flink/blob/master/azure-pipelines.yml which
references
https://github.com/apache/flink/blob/master/tools/azure-pipelines/jobs-template.yml
It uses the build matrix feature to parallelize the execution:
https://github.com/apache/flink/blob/dd0ee24e55dab4ae76201103c76495bc4fa0f73b/tools/azure-pipelines/jobs-template.yml#L88-L107
What would be the key benefit for Pulsar CI of using Azure Pipelines over
GitHub Actions?
-Lari
On Fri, Jan 29, 2021 at 6:03 PM Sijie Guo <gu...@gmail.com> wrote:
> Lari,
>
> Yes, we can keep this proposal open for discussion. That's for sure.
>
> I just don't have any good solution at this moment with a multiple-workflow
> approach using Github Actions.
>
> An alternative is to look into Azure Pipeline, which the Flink community is
> using.
> We are still learning there. Will post thoughts here once we have a better
> idea.
>
> Thanks,
> Sijie
>
> On Fri, Jan 29, 2021 at 5:07 AM Lari Hotari <La...@hotari.net> wrote:
>
> > Thanks for the feedback, Sijie.
> >
> > > If this proposal is blocked by the other proposal, we should focus on
> > getting the changes for the other proposal before talking about merging
> > them.
> >
> > Yes, the current proposal depends on the draft PIP for "Changes to flaky
> > test handling". I'll follow up on fixing the flaky test in a new email
> > thread.
> >
> > I hope we could get the discussions going on both draft PIPs and find
> > consensus together as a community.
> > During the discussions, more solution options will come up. Each solution
> > has trade offs.
> > It would be useful to document the options when the community doesn't
> > immediately agree on a single choice.
> > I was thinking that these options could be documented in the same draft
> PIP
> > documents.
> >
> > I can give multiple authors editing access to the Google Docs so that we
> > can keep on editing a single document for both draft PIPs.
> > Anyone who would want to add more solution options to the documents,
> please
> > let me know so that I'll add editing access.
> >
> > Sijie, would you like to document the option around keeping the workflow
> as
> > multiple smaller workflows?
> > I have understood that the problems that have come up with the Pulsar CI
> > regarding resource consumption would have to be resolved in that
> > alternative as well.
> >
> > I believe that everyone is open to any set of solution alternatives which
> > solves the problems that we have with Pulsar CI.
> > We all know that it's urgent to fix Pulsar CI asap. We can do it
> together.
> >
> > BR, Lari
> >
> >
> > On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <gu...@gmail.com> wrote:
> >
> > > Lari,
> > >
> > > Thank you for bringing this proposal up! This is a great initiative.
> > >
> > > However, I agreed with Yong. We have spent tons of effort splitting one
> > > large workflow into multiple smaller workflows.
> > >
> > > If this proposal is blocked by the other proposal, we should focus on
> > > getting the changes for the other proposal before talking about merging
> > > them.
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net> wrote:
> > >
> > > > Thank you for the comments Penghui.
> > > >
> > > > Exactly what you said, we should make the tests stable.
> > > > The proposals in the other draft PIP "Changes to flaky test handling"
> > > deals
> > > > with that.
> > > > It's currently a draft and needs more eyes. Would you be able to
> take a
> > > > closer look at that too?
> > > >
> > > > BR, Lari
> > > >
> > > > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <co...@gmail.com>
> > > > wrote:
> > > >
> > > > > Currently, especially for the integration tests, a lot time to
> build
> > > > > pulsar distributions and docker images.
> > > > > I think before merge tests we should to make the test stable,
> > otherwise
> > > > > rerun the test will become more expensive.
> > > > >
> > > > > Thanks,
> > > > > Penghui
> > > > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> > > zhangyong1025.zy@gmail.com
> > > > >,
> > > > > wrote:
> > > > > > I am not sure that merge all the workflows into one workflow is a
> > > good
> > > > > > idea. As
> > > > > > I know, Github Actions doesn't allow to rerun a single job in a
> > > > workflow.
> > > > > > That means
> > > > > > if there has any failure in the workflow, we need to rerun all
> > > > > > steps/stage. There has
> > > > > > a worst-case is we failed in the different tests when rerunning
> it
> > > and
> > > > > this
> > > > > > would take
> > > > > > more time to pass the CI.
> > > > > >
> > > > > > ---
> > > > > > Yong
> > > > > >
> > > > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <lari.hotari@sagire.fi
> >
> > > > wrote:
> > > > > >
> > > > > > > Dear Pulsar community members,
> > > > > > >
> > > > > > > Currently, the Pulsar GitHub Actions workflows are consuming
> the
> > > > > majority
> > > > > > > of the shared pool of resources allocated for
> github.com/apache
> > > > > projects.
> > > > > > > Other Apache projects have been impacted and there is a demand
> to
> > > > > improve
> > > > > > > the Pulsar CI
> > > > > > > <
> > https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > > >
> > > > > asap.
> > > > > > >
> > > > > > > In GitHub Actions Runners, the unit of resources is the time
> > that a
> > > > > Runner
> > > > > > > is occupied. I observed the workflow runs for handling a single
> > > Pull
> > > > > > > Request (in my personal fork) and these were the running
> > durations:
> > > > > > > Workflow name Duration
> > > > > > > CI - Build - MacOS 0:17:23
> > > > > > > CI - Go Functions style check 0:02:38
> > > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > > CI - Misc 0:16:51
> > > > > > > CI - Unit - Proxy 0:14:23
> > > > > > > CI - Go Functions Tests 0:22:08
> > > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > > CI - Unit 0:42:11
> > > > > > > CI - Integration - Sql 1:00:13
> > > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > > CI - Integration - Function State 1:00:12
> > > > > > > CI - Integration - Cli 1:10:22
> > > > > > > CI - Integration - Transaction 1:16:34
> > > > > > > CI - Integration - Process 1:11:23
> > > > > > > CI - Shade - Test 1:15:45
> > > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > > CI - Integration - Standalone 0:45:29
> > > > > > > CI - Integration - Messaging 1:00:23
> > > > > > > CI - Integration - Thread 1:00:19
> > > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > > CI - Integration - Schema 1:00:19
> > > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > > TOTAL 19:36:50
> > > > > > >
> > > > > > > *In this case, the total resource consumption of GitHub Actions
> > > > > Runners is
> > > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > > apache/pulsar.*
> > > > > > >
> > > > > > > Since GitHub Actions Runner resource pool utilization is very
> > high,
> > > > > this
> > > > > > > leads to the build queue to grow and take a long time to
> process.
> > > > > > >
> > > > > > > I have been looking for ways to improve the Pulsar CI for the
> > last
> > > 3
> > > > > > > months. During this period I worked on a few experiments. The
> > > > learnings
> > > > > > > from the past experiments are documented at a high level in the
> > > > > following
> > > > > > > draft PIP document.
> > > > > > >
> > > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> > document
> > > > is
> > > > > a
> > > > > > > Google doc:*
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > > >
> > > > > > > *Please participate* so that we get the plan adjusted based on
> > the
> > > > > feedback
> > > > > > > asap. If there's already a similar effort ongoing, I hope we
> can
> > > join
> > > > > > > efforts.
> > > > > > >
> > > > > > > *Let's fix Pulsar CI!*
> > > > > > >
> > > > > > > BR, Lari
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Sijie Guo <gu...@gmail.com>.
Lari,
Yes, we can keep this proposal open for discussion. That's for sure.
I just don't have any good solution at this moment with a multiple-workflow
approach using Github Actions.
An alternative is to look into Azure Pipeline, which the Flink community is
using.
We are still learning there. Will post thoughts here once we have a better
idea.
Thanks,
Sijie
On Fri, Jan 29, 2021 at 5:07 AM Lari Hotari <La...@hotari.net> wrote:
> Thanks for the feedback, Sijie.
>
> > If this proposal is blocked by the other proposal, we should focus on
> getting the changes for the other proposal before talking about merging
> them.
>
> Yes, the current proposal depends on the draft PIP for "Changes to flaky
> test handling". I'll follow up on fixing the flaky test in a new email
> thread.
>
> I hope we could get the discussions going on both draft PIPs and find
> consensus together as a community.
> During the discussions, more solution options will come up. Each solution
> has trade offs.
> It would be useful to document the options when the community doesn't
> immediately agree on a single choice.
> I was thinking that these options could be documented in the same draft PIP
> documents.
>
> I can give multiple authors editing access to the Google Docs so that we
> can keep on editing a single document for both draft PIPs.
> Anyone who would want to add more solution options to the documents, please
> let me know so that I'll add editing access.
>
> Sijie, would you like to document the option around keeping the workflow as
> multiple smaller workflows?
> I have understood that the problems that have come up with the Pulsar CI
> regarding resource consumption would have to be resolved in that
> alternative as well.
>
> I believe that everyone is open to any set of solution alternatives which
> solves the problems that we have with Pulsar CI.
> We all know that it's urgent to fix Pulsar CI asap. We can do it together.
>
> BR, Lari
>
>
> On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <gu...@gmail.com> wrote:
>
> > Lari,
> >
> > Thank you for bringing this proposal up! This is a great initiative.
> >
> > However, I agreed with Yong. We have spent tons of effort splitting one
> > large workflow into multiple smaller workflows.
> >
> > If this proposal is blocked by the other proposal, we should focus on
> > getting the changes for the other proposal before talking about merging
> > them.
> >
> > Thanks,
> > Sijie
> >
> > On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net> wrote:
> >
> > > Thank you for the comments Penghui.
> > >
> > > Exactly what you said, we should make the tests stable.
> > > The proposals in the other draft PIP "Changes to flaky test handling"
> > deals
> > > with that.
> > > It's currently a draft and needs more eyes. Would you be able to take a
> > > closer look at that too?
> > >
> > > BR, Lari
> > >
> > > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <co...@gmail.com>
> > > wrote:
> > >
> > > > Currently, especially for the integration tests, a lot time to build
> > > > pulsar distributions and docker images.
> > > > I think before merge tests we should to make the test stable,
> otherwise
> > > > rerun the test will become more expensive.
> > > >
> > > > Thanks,
> > > > Penghui
> > > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> > zhangyong1025.zy@gmail.com
> > > >,
> > > > wrote:
> > > > > I am not sure that merge all the workflows into one workflow is a
> > good
> > > > > idea. As
> > > > > I know, Github Actions doesn't allow to rerun a single job in a
> > > workflow.
> > > > > That means
> > > > > if there has any failure in the workflow, we need to rerun all
> > > > > steps/stage. There has
> > > > > a worst-case is we failed in the different tests when rerunning it
> > and
> > > > this
> > > > > would take
> > > > > more time to pass the CI.
> > > > >
> > > > > ---
> > > > > Yong
> > > > >
> > > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <la...@sagire.fi>
> > > wrote:
> > > > >
> > > > > > Dear Pulsar community members,
> > > > > >
> > > > > > Currently, the Pulsar GitHub Actions workflows are consuming the
> > > > majority
> > > > > > of the shared pool of resources allocated for github.com/apache
> > > > projects.
> > > > > > Other Apache projects have been impacted and there is a demand to
> > > > improve
> > > > > > the Pulsar CI
> > > > > > <
> https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> > >
> > > > asap.
> > > > > >
> > > > > > In GitHub Actions Runners, the unit of resources is the time
> that a
> > > > Runner
> > > > > > is occupied. I observed the workflow runs for handling a single
> > Pull
> > > > > > Request (in my personal fork) and these were the running
> durations:
> > > > > > Workflow name Duration
> > > > > > CI - Build - MacOS 0:17:23
> > > > > > CI - Go Functions style check 0:02:38
> > > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > > CI - Misc 0:16:51
> > > > > > CI - Unit - Proxy 0:14:23
> > > > > > CI - Go Functions Tests 0:22:08
> > > > > > CI - CPP, Python Tests 0:23:30
> > > > > > CI - Unit 0:42:11
> > > > > > CI - Integration - Sql 1:00:13
> > > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > > CI - Integration - Function State 1:00:12
> > > > > > CI - Integration - Cli 1:10:22
> > > > > > CI - Integration - Transaction 1:16:34
> > > > > > CI - Integration - Process 1:11:23
> > > > > > CI - Shade - Test 1:15:45
> > > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > > CI - Integration - Standalone 0:45:29
> > > > > > CI - Integration - Messaging 1:00:23
> > > > > > CI - Integration - Thread 1:00:19
> > > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > > CI - Integration - Schema 1:00:19
> > > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > > TOTAL 19:36:50
> > > > > >
> > > > > > *In this case, the total resource consumption of GitHub Actions
> > > > Runners is
> > > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > > apache/pulsar.*
> > > > > >
> > > > > > Since GitHub Actions Runner resource pool utilization is very
> high,
> > > > this
> > > > > > leads to the build queue to grow and take a long time to process.
> > > > > >
> > > > > > I have been looking for ways to improve the Pulsar CI for the
> last
> > 3
> > > > > > months. During this period I worked on a few experiments. The
> > > learnings
> > > > > > from the past experiments are documented at a high level in the
> > > > following
> > > > > > draft PIP document.
> > > > > >
> > > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI"
> document
> > > is
> > > > a
> > > > > > Google doc:*
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > > >
> > > > > > *Please participate* so that we get the plan adjusted based on
> the
> > > > feedback
> > > > > > asap. If there's already a similar effort ongoing, I hope we can
> > join
> > > > > > efforts.
> > > > > >
> > > > > > *Let's fix Pulsar CI!*
> > > > > >
> > > > > > BR, Lari
> > > > > >
> > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Thanks for the feedback, Sijie.
> If this proposal is blocked by the other proposal, we should focus on
getting the changes for the other proposal before talking about merging
them.
Yes, the current proposal depends on the draft PIP for "Changes to flaky
test handling". I'll follow up on fixing the flaky test in a new email
thread.
I hope we could get the discussions going on both draft PIPs and find
consensus together as a community.
During the discussions, more solution options will come up. Each solution
has trade offs.
It would be useful to document the options when the community doesn't
immediately agree on a single choice.
I was thinking that these options could be documented in the same draft PIP
documents.
I can give multiple authors editing access to the Google Docs so that we
can keep on editing a single document for both draft PIPs.
Anyone who would want to add more solution options to the documents, please
let me know so that I'll add editing access.
Sijie, would you like to document the option around keeping the workflow as
multiple smaller workflows?
I have understood that the problems that have come up with the Pulsar CI
regarding resource consumption would have to be resolved in that
alternative as well.
I believe that everyone is open to any set of solution alternatives which
solves the problems that we have with Pulsar CI.
We all know that it's urgent to fix Pulsar CI asap. We can do it together.
BR, Lari
On Fri, Jan 29, 2021 at 11:51 AM Sijie Guo <gu...@gmail.com> wrote:
> Lari,
>
> Thank you for bringing this proposal up! This is a great initiative.
>
> However, I agreed with Yong. We have spent tons of effort splitting one
> large workflow into multiple smaller workflows.
>
> If this proposal is blocked by the other proposal, we should focus on
> getting the changes for the other proposal before talking about merging
> them.
>
> Thanks,
> Sijie
>
> On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net> wrote:
>
> > Thank you for the comments Penghui.
> >
> > Exactly what you said, we should make the tests stable.
> > The proposals in the other draft PIP "Changes to flaky test handling"
> deals
> > with that.
> > It's currently a draft and needs more eyes. Would you be able to take a
> > closer look at that too?
> >
> > BR, Lari
> >
> > On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <co...@gmail.com>
> > wrote:
> >
> > > Currently, especially for the integration tests, a lot time to build
> > > pulsar distributions and docker images.
> > > I think before merge tests we should to make the test stable, otherwise
> > > rerun the test will become more expensive.
> > >
> > > Thanks,
> > > Penghui
> > > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <
> zhangyong1025.zy@gmail.com
> > >,
> > > wrote:
> > > > I am not sure that merge all the workflows into one workflow is a
> good
> > > > idea. As
> > > > I know, Github Actions doesn't allow to rerun a single job in a
> > workflow.
> > > > That means
> > > > if there has any failure in the workflow, we need to rerun all
> > > > steps/stage. There has
> > > > a worst-case is we failed in the different tests when rerunning it
> and
> > > this
> > > > would take
> > > > more time to pass the CI.
> > > >
> > > > ---
> > > > Yong
> > > >
> > > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <la...@sagire.fi>
> > wrote:
> > > >
> > > > > Dear Pulsar community members,
> > > > >
> > > > > Currently, the Pulsar GitHub Actions workflows are consuming the
> > > majority
> > > > > of the shared pool of resources allocated for github.com/apache
> > > projects.
> > > > > Other Apache projects have been impacted and there is a demand to
> > > improve
> > > > > the Pulsar CI
> > > > > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396
> >
> > > asap.
> > > > >
> > > > > In GitHub Actions Runners, the unit of resources is the time that a
> > > Runner
> > > > > is occupied. I observed the workflow runs for handling a single
> Pull
> > > > > Request (in my personal fork) and these were the running durations:
> > > > > Workflow name Duration
> > > > > CI - Build - MacOS 0:17:23
> > > > > CI - Go Functions style check 0:02:38
> > > > > CI - Unit - Brokers - Other 0:15:40
> > > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > > CI - Misc 0:16:51
> > > > > CI - Unit - Proxy 0:14:23
> > > > > CI - Go Functions Tests 0:22:08
> > > > > CI - CPP, Python Tests 0:23:30
> > > > > CI - Unit 0:42:11
> > > > > CI - Integration - Sql 1:00:13
> > > > > CI - Integration - Tiered JCloud 1:00:18
> > > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > > CI - Integration - Function State 1:00:12
> > > > > CI - Integration - Cli 1:10:22
> > > > > CI - Integration - Transaction 1:16:34
> > > > > CI - Integration - Process 1:11:23
> > > > > CI - Shade - Test 1:15:45
> > > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > > CI - Integration - Standalone 0:45:29
> > > > > CI - Integration - Messaging 1:00:23
> > > > > CI - Integration - Thread 1:00:19
> > > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > > CI - Integration - Schema 1:00:19
> > > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > > TOTAL 19:36:50
> > > > >
> > > > > *In this case, the total resource consumption of GitHub Actions
> > > Runners is
> > > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > > apache/pulsar.*
> > > > >
> > > > > Since GitHub Actions Runner resource pool utilization is very high,
> > > this
> > > > > leads to the build queue to grow and take a long time to process.
> > > > >
> > > > > I have been looking for ways to improve the Pulsar CI for the last
> 3
> > > > > months. During this period I worked on a few experiments. The
> > learnings
> > > > > from the past experiments are documented at a high level in the
> > > following
> > > > > draft PIP document.
> > > > >
> > > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document
> > is
> > > a
> > > > > Google doc:*
> > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > > >
> > > > > *Please participate* so that we get the plan adjusted based on the
> > > feedback
> > > > > asap. If there's already a similar effort ongoing, I hope we can
> join
> > > > > efforts.
> > > > >
> > > > > *Let's fix Pulsar CI!*
> > > > >
> > > > > BR, Lari
> > > > >
> > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Sijie Guo <gu...@gmail.com>.
Lari,
Thank you for bringing this proposal up! This is a great initiative.
However, I agreed with Yong. We have spent tons of effort splitting one
large workflow into multiple smaller workflows.
If this proposal is blocked by the other proposal, we should focus on
getting the changes for the other proposal before talking about merging
them.
Thanks,
Sijie
On Thu, Jan 28, 2021 at 9:55 PM Lari Hotari <La...@hotari.net> wrote:
> Thank you for the comments Penghui.
>
> Exactly what you said, we should make the tests stable.
> The proposals in the other draft PIP "Changes to flaky test handling" deals
> with that.
> It's currently a draft and needs more eyes. Would you be able to take a
> closer look at that too?
>
> BR, Lari
>
> On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <co...@gmail.com>
> wrote:
>
> > Currently, especially for the integration tests, a lot time to build
> > pulsar distributions and docker images.
> > I think before merge tests we should to make the test stable, otherwise
> > rerun the test will become more expensive.
> >
> > Thanks,
> > Penghui
> > On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <zhangyong1025.zy@gmail.com
> >,
> > wrote:
> > > I am not sure that merge all the workflows into one workflow is a good
> > > idea. As
> > > I know, Github Actions doesn't allow to rerun a single job in a
> workflow.
> > > That means
> > > if there has any failure in the workflow, we need to rerun all
> > > steps/stage. There has
> > > a worst-case is we failed in the different tests when rerunning it and
> > this
> > > would take
> > > more time to pass the CI.
> > >
> > > ---
> > > Yong
> > >
> > > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <la...@sagire.fi>
> wrote:
> > >
> > > > Dear Pulsar community members,
> > > >
> > > > Currently, the Pulsar GitHub Actions workflows are consuming the
> > majority
> > > > of the shared pool of resources allocated for github.com/apache
> > projects.
> > > > Other Apache projects have been impacted and there is a demand to
> > improve
> > > > the Pulsar CI
> > > > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396>
> > asap.
> > > >
> > > > In GitHub Actions Runners, the unit of resources is the time that a
> > Runner
> > > > is occupied. I observed the workflow runs for handling a single Pull
> > > > Request (in my personal fork) and these were the running durations:
> > > > Workflow name Duration
> > > > CI - Build - MacOS 0:17:23
> > > > CI - Go Functions style check 0:02:38
> > > > CI - Unit - Brokers - Other 0:15:40
> > > > CI - Unit - Brokers - Client Impl 0:16:28
> > > > CI - Misc 0:16:51
> > > > CI - Unit - Proxy 0:14:23
> > > > CI - Go Functions Tests 0:22:08
> > > > CI - CPP, Python Tests 0:23:30
> > > > CI - Unit 0:42:11
> > > > CI - Integration - Sql 1:00:13
> > > > CI - Integration - Tiered JCloud 1:00:18
> > > > CI - Integration - Tiered FileSystem 1:00:13
> > > > CI - Integration - Function State 1:00:12
> > > > CI - Integration - Cli 1:10:22
> > > > CI - Integration - Transaction 1:16:34
> > > > CI - Integration - Process 1:11:23
> > > > CI - Shade - Test 1:15:45
> > > > CI - Unit - Brokers - Client Api 0:26:13
> > > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > > CI - Integration - Standalone 0:45:29
> > > > CI - Integration - Messaging 1:00:23
> > > > CI - Integration - Thread 1:00:19
> > > > CI - Integration - Backwards Compatibility 1:00:19
> > > > CI - Integration - Schema 1:00:19
> > > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > > TOTAL 19:36:50
> > > >
> > > > *In this case, the total resource consumption of GitHub Actions
> > Runners is
> > > > 19 hours 36 minutes 50 seconds for a single pull request to
> > apache/pulsar.*
> > > >
> > > > Since GitHub Actions Runner resource pool utilization is very high,
> > this
> > > > leads to the build queue to grow and take a long time to process.
> > > >
> > > > I have been looking for ways to improve the Pulsar CI for the last 3
> > > > months. During this period I worked on a few experiments. The
> learnings
> > > > from the past experiments are documented at a high level in the
> > following
> > > > draft PIP document.
> > > >
> > > > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document
> is
> > a
> > > > Google doc:*
> > > >
> > > >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > > >
> > > > *Please participate* so that we get the plan adjusted based on the
> > feedback
> > > > asap. If there's already a similar effort ongoing, I hope we can join
> > > > efforts.
> > > >
> > > > *Let's fix Pulsar CI!*
> > > >
> > > > BR, Lari
> > > >
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Thank you for the comments Penghui.
Exactly what you said, we should make the tests stable.
The proposals in the other draft PIP "Changes to flaky test handling" deals
with that.
It's currently a draft and needs more eyes. Would you be able to take a
closer look at that too?
BR, Lari
On Fri, Jan 29, 2021 at 6:41 AM PengHui Li <co...@gmail.com> wrote:
> Currently, especially for the integration tests, a lot time to build
> pulsar distributions and docker images.
> I think before merge tests we should to make the test stable, otherwise
> rerun the test will become more expensive.
>
> Thanks,
> Penghui
> On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <zh...@gmail.com>,
> wrote:
> > I am not sure that merge all the workflows into one workflow is a good
> > idea. As
> > I know, Github Actions doesn't allow to rerun a single job in a workflow.
> > That means
> > if there has any failure in the workflow, we need to rerun all
> > steps/stage. There has
> > a worst-case is we failed in the different tests when rerunning it and
> this
> > would take
> > more time to pass the CI.
> >
> > ---
> > Yong
> >
> > On Fri, 29 Jan 2021 at 01:14, Lari Hotari <la...@sagire.fi> wrote:
> >
> > > Dear Pulsar community members,
> > >
> > > Currently, the Pulsar GitHub Actions workflows are consuming the
> majority
> > > of the shared pool of resources allocated for github.com/apache
> projects.
> > > Other Apache projects have been impacted and there is a demand to
> improve
> > > the Pulsar CI
> > > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396>
> asap.
> > >
> > > In GitHub Actions Runners, the unit of resources is the time that a
> Runner
> > > is occupied. I observed the workflow runs for handling a single Pull
> > > Request (in my personal fork) and these were the running durations:
> > > Workflow name Duration
> > > CI - Build - MacOS 0:17:23
> > > CI - Go Functions style check 0:02:38
> > > CI - Unit - Brokers - Other 0:15:40
> > > CI - Unit - Brokers - Client Impl 0:16:28
> > > CI - Misc 0:16:51
> > > CI - Unit - Proxy 0:14:23
> > > CI - Go Functions Tests 0:22:08
> > > CI - CPP, Python Tests 0:23:30
> > > CI - Unit 0:42:11
> > > CI - Integration - Sql 1:00:13
> > > CI - Integration - Tiered JCloud 1:00:18
> > > CI - Integration - Tiered FileSystem 1:00:13
> > > CI - Integration - Function State 1:00:12
> > > CI - Integration - Cli 1:10:22
> > > CI - Integration - Transaction 1:16:34
> > > CI - Integration - Process 1:11:23
> > > CI - Shade - Test 1:15:45
> > > CI - Unit - Brokers - Client Api 0:26:13
> > > CI - Unit - Brokers - Broker Group 2 0:35:05
> > > CI - Integration - Standalone 0:45:29
> > > CI - Integration - Messaging 1:00:23
> > > CI - Integration - Thread 1:00:19
> > > CI - Integration - Backwards Compatibility 1:00:19
> > > CI - Integration - Schema 1:00:19
> > > CI - Unit - Brokers - Broker Group 1 2:02:31
> > > TOTAL 19:36:50
> > >
> > > *In this case, the total resource consumption of GitHub Actions
> Runners is
> > > 19 hours 36 minutes 50 seconds for a single pull request to
> apache/pulsar.*
> > >
> > > Since GitHub Actions Runner resource pool utilization is very high,
> this
> > > leads to the build queue to grow and take a long time to process.
> > >
> > > I have been looking for ways to improve the Pulsar CI for the last 3
> > > months. During this period I worked on a few experiments. The learnings
> > > from the past experiments are documented at a high level in the
> following
> > > draft PIP document.
> > >
> > > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document is
> a
> > > Google doc:*
> > >
> > >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> > >
> > > *Please participate* so that we get the plan adjusted based on the
> feedback
> > > asap. If there's already a similar effort ongoing, I hope we can join
> > > efforts.
> > >
> > > *Let's fix Pulsar CI!*
> > >
> > > BR, Lari
> > >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based
Pulsar CI"
Posted by PengHui Li <co...@gmail.com>.
Currently, especially for the integration tests, a lot time to build pulsar distributions and docker images.
I think before merge tests we should to make the test stable, otherwise rerun the test will become more expensive.
Thanks,
Penghui
On Jan 29, 2021, 11:55 AM +0800, Yong Zhang <zh...@gmail.com>, wrote:
> I am not sure that merge all the workflows into one workflow is a good
> idea. As
> I know, Github Actions doesn't allow to rerun a single job in a workflow.
> That means
> if there has any failure in the workflow, we need to rerun all
> steps/stage. There has
> a worst-case is we failed in the different tests when rerunning it and this
> would take
> more time to pass the CI.
>
> ---
> Yong
>
> On Fri, 29 Jan 2021 at 01:14, Lari Hotari <la...@sagire.fi> wrote:
>
> > Dear Pulsar community members,
> >
> > Currently, the Pulsar GitHub Actions workflows are consuming the majority
> > of the shared pool of resources allocated for github.com/apache projects.
> > Other Apache projects have been impacted and there is a demand to improve
> > the Pulsar CI
> > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396> asap.
> >
> > In GitHub Actions Runners, the unit of resources is the time that a Runner
> > is occupied. I observed the workflow runs for handling a single Pull
> > Request (in my personal fork) and these were the running durations:
> > Workflow name Duration
> > CI - Build - MacOS 0:17:23
> > CI - Go Functions style check 0:02:38
> > CI - Unit - Brokers - Other 0:15:40
> > CI - Unit - Brokers - Client Impl 0:16:28
> > CI - Misc 0:16:51
> > CI - Unit - Proxy 0:14:23
> > CI - Go Functions Tests 0:22:08
> > CI - CPP, Python Tests 0:23:30
> > CI - Unit 0:42:11
> > CI - Integration - Sql 1:00:13
> > CI - Integration - Tiered JCloud 1:00:18
> > CI - Integration - Tiered FileSystem 1:00:13
> > CI - Integration - Function State 1:00:12
> > CI - Integration - Cli 1:10:22
> > CI - Integration - Transaction 1:16:34
> > CI - Integration - Process 1:11:23
> > CI - Shade - Test 1:15:45
> > CI - Unit - Brokers - Client Api 0:26:13
> > CI - Unit - Brokers - Broker Group 2 0:35:05
> > CI - Integration - Standalone 0:45:29
> > CI - Integration - Messaging 1:00:23
> > CI - Integration - Thread 1:00:19
> > CI - Integration - Backwards Compatibility 1:00:19
> > CI - Integration - Schema 1:00:19
> > CI - Unit - Brokers - Broker Group 1 2:02:31
> > TOTAL 19:36:50
> >
> > *In this case, the total resource consumption of GitHub Actions Runners is
> > 19 hours 36 minutes 50 seconds for a single pull request to apache/pulsar.*
> >
> > Since GitHub Actions Runner resource pool utilization is very high, this
> > leads to the build queue to grow and take a long time to process.
> >
> > I have been looking for ways to improve the Pulsar CI for the last 3
> > months. During this period I worked on a few experiments. The learnings
> > from the past experiments are documented at a high level in the following
> > draft PIP document.
> >
> > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document is a
> > Google doc:*
> >
> > https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> >
> > *Please participate* so that we get the plan adjusted based on the feedback
> > asap. If there's already a similar effort ongoing, I hope we can join
> > efforts.
> >
> > *Let's fix Pulsar CI!*
> >
> > BR, Lari
> >
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Lari Hotari <La...@hotari.net>.
Thanks for the comments, Yong.
Very good point. Re-running a single job isn't supported.
Currently, re-rerunning is required because it is the way we deal with
flaky tests.
There is a separate draft PIP for "Changes to flaky test handling" in
discussion,
https://lists.apache.org/thread.html/r4c23782160f4ac1c3e2533b5f767a8e0d3ef934820c3ceb6b3925918%40%3Cdev.pulsar.apache.org%3E
.
The goal is to get the flaky test problem addressed so that re-running a
workflow would not be required.
BR, Lari
On Fri, Jan 29, 2021 at 5:55 AM Yong Zhang <zh...@gmail.com>
wrote:
> I am not sure that merge all the workflows into one workflow is a good
> idea. As
> I know, Github Actions doesn't allow to rerun a single job in a workflow.
> That means
> if there has any failure in the workflow, we need to rerun all
> steps/stage. There has
> a worst-case is we failed in the different tests when rerunning it and this
> would take
> more time to pass the CI.
>
> ---
> Yong
>
> On Fri, 29 Jan 2021 at 01:14, Lari Hotari <la...@sagire.fi> wrote:
>
> > Dear Pulsar community members,
> >
> > Currently, the Pulsar GitHub Actions workflows are consuming the majority
> > of the shared pool of resources allocated for github.com/apache
> projects.
> > Other Apache projects have been impacted and there is a demand to improve
> > the Pulsar CI
> > <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396>
> asap.
> >
> > In GitHub Actions Runners, the unit of resources is the time that a
> Runner
> > is occupied. I observed the workflow runs for handling a single Pull
> > Request (in my personal fork) and these were the running durations:
> > Workflow name Duration
> > CI - Build - MacOS 0:17:23
> > CI - Go Functions style check 0:02:38
> > CI - Unit - Brokers - Other 0:15:40
> > CI - Unit - Brokers - Client Impl 0:16:28
> > CI - Misc 0:16:51
> > CI - Unit - Proxy 0:14:23
> > CI - Go Functions Tests 0:22:08
> > CI - CPP, Python Tests 0:23:30
> > CI - Unit 0:42:11
> > CI - Integration - Sql 1:00:13
> > CI - Integration - Tiered JCloud 1:00:18
> > CI - Integration - Tiered FileSystem 1:00:13
> > CI - Integration - Function State 1:00:12
> > CI - Integration - Cli 1:10:22
> > CI - Integration - Transaction 1:16:34
> > CI - Integration - Process 1:11:23
> > CI - Shade - Test 1:15:45
> > CI - Unit - Brokers - Client Api 0:26:13
> > CI - Unit - Brokers - Broker Group 2 0:35:05
> > CI - Integration - Standalone 0:45:29
> > CI - Integration - Messaging 1:00:23
> > CI - Integration - Thread 1:00:19
> > CI - Integration - Backwards Compatibility 1:00:19
> > CI - Integration - Schema 1:00:19
> > CI - Unit - Brokers - Broker Group 1 2:02:31
> > TOTAL 19:36:50
> >
> > *In this case, the total resource consumption of GitHub Actions Runners
> is
> > 19 hours 36 minutes 50 seconds for a single pull request to
> apache/pulsar.*
> >
> > Since GitHub Actions Runner resource pool utilization is very high, this
> > leads to the build queue to grow and take a long time to process.
> >
> > I have been looking for ways to improve the Pulsar CI for the last 3
> > months. During this period I worked on a few experiments. The learnings
> > from the past experiments are documented at a high level in the following
> > draft PIP document.
> >
> > *The draft PIP "Changes to GitHub Actions based Pulsar CI" document is a
> > Google doc:*
> >
> >
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
> >
> > *Please participate* so that we get the plan adjusted based on the
> feedback
> > asap. If there's already a similar effort ongoing, I hope we can join
> > efforts.
> >
> > *Let's fix Pulsar CI!*
> >
> > BR, Lari
> >
>
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
Posted by Yong Zhang <zh...@gmail.com>.
I am not sure that merge all the workflows into one workflow is a good
idea. As
I know, Github Actions doesn't allow to rerun a single job in a workflow.
That means
if there has any failure in the workflow, we need to rerun all
steps/stage. There has
a worst-case is we failed in the different tests when rerunning it and this
would take
more time to pass the CI.
---
Yong
On Fri, 29 Jan 2021 at 01:14, Lari Hotari <la...@sagire.fi> wrote:
> Dear Pulsar community members,
>
> Currently, the Pulsar GitHub Actions workflows are consuming the majority
> of the shared pool of resources allocated for github.com/apache projects.
> Other Apache projects have been impacted and there is a demand to improve
> the Pulsar CI
> <https://github.com/apache/pulsar/pull/9159#issuecomment-766915396> asap.
>
> In GitHub Actions Runners, the unit of resources is the time that a Runner
> is occupied. I observed the workflow runs for handling a single Pull
> Request (in my personal fork) and these were the running durations:
> Workflow name Duration
> CI - Build - MacOS 0:17:23
> CI - Go Functions style check 0:02:38
> CI - Unit - Brokers - Other 0:15:40
> CI - Unit - Brokers - Client Impl 0:16:28
> CI - Misc 0:16:51
> CI - Unit - Proxy 0:14:23
> CI - Go Functions Tests 0:22:08
> CI - CPP, Python Tests 0:23:30
> CI - Unit 0:42:11
> CI - Integration - Sql 1:00:13
> CI - Integration - Tiered JCloud 1:00:18
> CI - Integration - Tiered FileSystem 1:00:13
> CI - Integration - Function State 1:00:12
> CI - Integration - Cli 1:10:22
> CI - Integration - Transaction 1:16:34
> CI - Integration - Process 1:11:23
> CI - Shade - Test 1:15:45
> CI - Unit - Brokers - Client Api 0:26:13
> CI - Unit - Brokers - Broker Group 2 0:35:05
> CI - Integration - Standalone 0:45:29
> CI - Integration - Messaging 1:00:23
> CI - Integration - Thread 1:00:19
> CI - Integration - Backwards Compatibility 1:00:19
> CI - Integration - Schema 1:00:19
> CI - Unit - Brokers - Broker Group 1 2:02:31
> TOTAL 19:36:50
>
> *In this case, the total resource consumption of GitHub Actions Runners is
> 19 hours 36 minutes 50 seconds for a single pull request to apache/pulsar.*
>
> Since GitHub Actions Runner resource pool utilization is very high, this
> leads to the build queue to grow and take a long time to process.
>
> I have been looking for ways to improve the Pulsar CI for the last 3
> months. During this period I worked on a few experiments. The learnings
> from the past experiments are documented at a high level in the following
> draft PIP document.
>
> *The draft PIP "Changes to GitHub Actions based Pulsar CI" document is a
> Google doc:*
>
> https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing
>
> *Please participate* so that we get the plan adjusted based on the feedback
> asap. If there's already a similar effort ongoing, I hope we can join
> efforts.
>
> *Let's fix Pulsar CI!*
>
> BR, Lari
>