[jira] [Created] (FLINK-40248) Move publishing nightly snapshots from AZP to GHA

["Dale Lane (Jira)" <ji...@apache.org>]

Dale Lane created FLINK-40248:
---------------------------------

             Summary: Move publishing nightly snapshots from AZP to GHA
                 Key: FLINK-40248
                 URL: https://issues.apache.org/jira/browse/FLINK-40248
             Project: Flink
          Issue Type: Sub-task
          Components: Build System / CI
            Reporter: Dale Lane


Today, the nightly scheduled Azure Pipelines pipeline:
1) uploads release binaries to S3
2) pushes snapshot jars to Maven

Before we could migrate fully off of AZP, we need an equivalent nightly scheduled job that does this from Github Actions. 

We'll do this in a phased way, one release branch at a time.

h3. Approach

Add the equivalent snapshot publishing to the GHA nightly workflow, covering both the binary release upload and the Maven snapshot deploy.

Which pipeline publishes a given branch will be configuration held in the GHA nightly workflow itself. 

Because that workflow is versioned per branch, each branch can be switched over independently, and a branch that hasn't been switched over yet still *builds* its snapshots on GHA (it just keeps them as build artifacts of the workflow run rather than publishing them). 

The aim of this is to make sure that the new GHA jobs get exercised on every branch well before they become the canonical source of anything we  publish.

The safety property we want throughout is that exactly one pipeline is publishing a given branch at any time. Two pipelines publishing the same branch would overwrite each other (neither publishing would arguably be worse, as a branch's snapshots would silently go stale). 

For each branch, enabling GHA and disabling AZP should be done together, as one reviewable change.

h3. Rollout

The first branch that we migrate should have an extra safety net step that let's us compare GHA's output against AZP's output on a few real nightly runs before anything is depending on GHA output. 

# *Both pipelines publish the branch.* GHA publishes to a separate location that nothing consumes, so it can't overwrite what AZP is publishing. We compare the two sets of output over a series of nightly runs to confirm parity.
# *Hand the branch over.* We make the switch described above, so GHA starts publishing to the real location and AZP stops publishing the branch. We then verify the published snapshots directly.
# *Migrate the remaining branches.* With the approach proven end to end, each subsequent branch is switched over in a single step - no shadow phase needed.

h3. Not in scope

I'm not talking about migrating the nightly *test* jobs here, or the manual release process. This is only about the nightly snapshot builds currently produced from AZP 


h3. Done when

This subtask will be complete once every active release branch publishes its nightly snapshots from GHA, and AZP no longer does. At that point, the  nightly snapshot deployment configuration can be deleted from the AZP pipeline



--
This message was sent by Atlassian Jira
(v8.20.10#820010)