[GitHub] [airflow] potiuk commented on a change in pull request #8898: Correct generation and package generation scripts

[GitBox <gi...@apache.org>]

potiuk commented on a change in pull request #8898:
URL: https://github.com/apache/airflow/pull/8898#discussion_r427236160



##########
File path: breeze
##########
@@ -1629,9 +1647,13 @@ function flag_push_docker_images() {
 # Prints flags that control version of generated packages
 function flag_version_suffix() {
       echo "
--S, --version-suffix
-        Adds optional suffix to the generated backport package version. It can be used to generate
-        rc1/rc2 ... versions of the packages.
+-S, --version-suffix-for-pypi
+        Adds optional suffix to the version in the generated backport package. It can be used
+        to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.

Review comment:
       We don't have to. Let me explain.
   
   When we use the `--version-suffix-for-svn rc1` we generate packages without any suffix and we rename them to contain the rc1 in the name. This is so that when we release we can rename the packages back.
   
   However when you use the `--version-suffix-for-pypi`, the suffix is added dynamically during package generation - it's appended to the version of package. 
   
   The version is not read from version.py, it is read from the most recent PROVIDERS_CHANGES_YYYY_MM_DD.md  -> YYYY.[M]M.[D]D 
   
   Theoretically, it can be a different version for each package being prepared (depending which PROVIDERS_CHANGES_YYYY_MM_DD.md is most recent (by the date). 
   
   It works like this:
   
   1) 
   ```
   ./breeze prepare-backport-readme -- 2020.05.20 PACKAGE1 PACKAGE2
   ```
   
   Prepares the README.md and PROVIDERS_CHANGES_2020.05.20.md for the packages specified.
   
   You commit that and only tag the release after it has been merged into master.
   
   2) Then from that tagged release.  when you want to generate those packages you run:
   
   ```
   ./breeze prepare-backport-packages --version-suffix-for-pypi rc1 -- PACKAGE1 PACKAGE2
   ```
   and the packages are generated with 2020.5.20rc1 version. Under the hood `setup_backport_packages.py` is run with --version-suffix rc1 flag and it will simply append the suffix to the  package version. There is no need to modify any sources temporarily, the suffix is added dynamically via result o this method call:
   
   ```
   def get_package_release_version(provider_package_id: str, version_suffix: str = "") -> str:
       """
       Returns release version including optional suffix.
   
       :param provider_package_id: package id
       :param version_suffix: optional suffix (rc1, rc2 etc).
       :return:
       """
       return get_latest_release(
           get_package_path(provider_package_id=provider_package_id)).release_version + version_suffix
   ```
   
   3) Later when you want to release final packages for PyPI you run (from the same tag/commit) :
   
   ```
   ./breeze prepare-backport-packages -- PACKAGE1 PACKAGE2
   ```
   And no version suffix is added in this case.
   
   4) For SVN packages you simply rename the packages already in SVN. They do not contain the rc1 suffix internally - only in the name.
   
   Currently, I am only using it to generate all packages so there is no risk of mistakes but later I want to add additional switch `prepare-backport-packages --package-version 2020.05.20`. This will verify if the packages being released have all the 'CHANGES' prepared (via `prepare-backport-readme`)  and that they are the latest ''CHANGES' and that this version has not already been released (checking for tag version in the git log and possibly pypi). But I will only be able to test it after we release officially the first wave of packages so I wanted to implement it later. This will protect against the accidental generation of the same package version as already released. 
   
   Eventually, I want to get rid of the manual operations and have all the steps automated within breeze so that you do not have to copy&paste commands from BACKPORT_PACKAGES.md, but this will require at least few iterations of releasing packages to get it consistently right.
   
   I want the release process to take a few minutes and have it fully repeatable by anyone without setting up anything else but the local key and pypirc configuration. I want to be able to release quite a bit more often in the future - especially if we go AIP-8 route and split Airflow 2.0 into separate "core" and "providers" packages from day one. I treat backport packages as a testing ground for that approach to see if it is workable, so that I can propose it.
   
   We can likely adopt it later to release airflow in the same way - without having to modify and discard version.py and add breeze command to do the same - but I planned to do it later.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org