You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/23 22:28:00 UTC

[GitHub] [airflow] turbaszek opened a new issue #11112: Release airflow upgrade_check as separate package

turbaszek opened a new issue #11112:
URL: https://github.com/apache/airflow/issues/11112


   **Description**
   
   Release `airflow upgrade_check` as an installable package in addition to command in 1.10.13/14 release.
   
   **Use case / motivation**
   
   In this way, users won't have to upgrade Airflow version only to use this tool. The upgrade check currently should be compatible with any 1.10.X version.
   
   Users that will use this tool have already installed Airflow and doing `pip install airflow_upgrade_check` should be not a problem.
   
   **Related Issues**
   
   N/A
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-720730714

Comment from my side as promised:

I think we can use a similar process I implemented for docker image separation. I have a working solution here: https://github.com/potiuk/airflow-docker

The "main" branch there is just synchronizing the repo. The "master" and "v1-10-stable" are just mirrors of "master" and "v1-10-stable" from airflow (filtered to only contain The Dockerfile + relevant files). It extracts only the relevant paths/commits. So if we structure the "upgrade-check" properly, it can extract only the "upgrade-check" relevant stuff to this repo.

Benefits:

* We do not have to manually synchronize v1-10 airflow code to such a project (it happens automatically).
* We can continue running full tests and whole CI checks in airflow - where we have all the pre-commits, tests, etc.
* The synchronization is scheduled and runs in "pull" mode from airflow's public repo, so there is no need for additional authentication (and secrets).

Caveats:
* for now the "main" branch contains the .github/workflow schedule and the code that synchronizes the repos. It does not contain the actual Dockerfile + scripts. This is because the scheduled workflows must reside in the "main" branch and I did not want to mix the "sync" code with the actual Dockerfiles. However, it should be entirely possible to have a single "main branch" containing both the "sync" code and the "upgrade-check" code.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

kaxil commented on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-724198428


   Any updates on this one @turbaszek @dimberman ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

mik-laj commented on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-698009179


   This may be a problem with database migrations. This tool would then have to contain copies of the newest migrations.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] vikramkoka commented on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

vikramkoka commented on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-720676788


   Just to make sure that I understand the scope of this clearly, does this include the following elements:
   
   1. Upgrade checks to be run from 1.10.x to say if this is compatible with 2.0, 
   2. Documentation to support above and including the overall guide for upgrading to 2.0, and
   3. Upgrade utils to automate as much as possible of the migration steps. 
   
   Is this understanding correct? 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

ashb commented on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-731445394


   Release created, but still some tidy up of docs/process left to do so it is repeatable.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] turbaszek commented on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

turbaszek commented on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-735226357


   @ashb should we close the issue?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-720730714






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb closed issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

ashb closed issue #11112:
URL: https://github.com/apache/airflow/issues/11112


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

kaxil commented on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-730463856

**Meeting - 19 Nov 2020**

Attendees: Ash, Vikram, Jarek, Tomasz, Daniel, Kaxil

- We will continue using v1-10-stable for upgrade_checks and the PRs targetted towards it.
- We will support Airflow >=1.10.13 for upgrade_checks.
- CI needs to be updated to test/support multiple versions.
- Idea: Add a "version" field for checks to show supported Airflow version.

- Ash to work on packaging and automation this/next week.
- Add a check when using the plugin to see if a newer version of upgrade_check is available. Error / Warning ??? (This can be a Rule in upgrade-check itself)

- If we include "--apply" flag that automatically fixes the code, we need to flag and WARN that to users that there will be **False positives** as Bowler is known to fail in some-cases (TypeHints). Have users confirm before apply any change and show the code diff that is being changed -- same as what Bowler does.

- Where we don't automate, just verify we have it documented

Release Management:
- Have setup.py / setup.cfg inside airflow/upgrade_check folder (Ash to figure out details)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #11112: Release airflow upgrade_check as separate package

Posted by GitBox <gi...@apache.org>.

ashb commented on issue #11112:
URL: https://github.com/apache/airflow/issues/11112#issuecomment-730592209


   I'm working on the release process part for this now.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org