You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/04 09:57:18 UTC
[GitHub] [airflow] mik-laj opened a new issue #10152: Release Airflow 2.0
mik-laj opened a new issue #10152:
URL: https://github.com/apache/airflow/issues/10152
Dear committer/PMC member,
This type of ticket is something new that they want to discuss on the mailing list, but they have to create one to be able to show it.
Dear user,
This ticket is in the POC of the new kind of tickets and may not contain up-to-date information. If you came here by accident, please ignore it.
----
Hello,
I am creating this ticket to help follow this process and coordinate our activities. This is a meta-ticket, so I would ask committers and PMC members to keep it up-to-date so that everyone has easy access to up-to-date information. This ticket should not describe individual changes to the code but summarize information that active community members know, but new community members don't know but want to know. This will make it more transparent and friendly.
# Release date/scope
We need to specify the release date and determine the scope of work. The scope of work is described in the [roadmap](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0).
Link: https://lists.apache.org/thread.html/r0abba3669962f101d787ad793611ba436d35c8e022aa565705778b7d%40%3Cdev.airflow.apache.org%3E
Status: **WIP**
# Complete the roadmap
Apart from defining the scope of work, we should also complete these tasks.
**High-level features**
Feaature | Status | More information
-- | -- | --
KEDA | Done | ?
Improve Webserver performance | ? |
Enhanced real-time UI | ? |
Improve Scheduler performance and reliability | ? |
Extend/finish the API | WIP | [Link](https://github.com/apache/airflow/issues/8107)
Production Docker image | WIP | [Project](https://github.com/apache/airflow/projects/3)
Production Helm chart | ? | ?
**AIPs**
| Feaature | Statuss | Meta-issue |
|-|-|-|
| [AIP-10: Multi-layered and multi-stage official Airflow CI image](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+CI+image) | ✅ | ? |
| [AIP-21: Changes in import paths](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths) | ✅ | |
| [AIP-16: Use nested commands in CLI](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-16%3A+Use+nested+commands+in+CLI) | ✅ | |
| [AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651) | WIP | |
| [AIP-32: Airflow REST API](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-32%3A+Airflow+REST+API) | WIP | [Link](https://github.com/apache/airflow/issues/8107) |
| [AIP-26 Production-ready Airflow Docker Image and helm chart](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+Production-ready+Airflow+Docker+Image+and+helm+chart) | WIP | |
| [AIP-3 Drop support for Python 2](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-3+Drop+support+for+Python+2) | ✅ | |
# Migration to Airflow 2.0
The 2.0 release of the Airflow is a significant upgrade and includes substantial major changes, For this reason, we must take additional steps to facilitate migration.
We discussed the migration path that a user should do in order to start using Airflow 2.0 easily
1. When the user is using Python 2.7, they need to migrate to Python 3.6+.
2. When the user is using Python 3.6+, they can start to use new operators and hooks from Python 3.6 thanks to backport packages.
3. When the user is on the latest Airflow 1.10 release, they can use the `airflow upgrade-check` command to see if they can migrate to the new Airflow version. This command checks for common incompatibility problems.
4. In the next step, the user installs the latest version of Airflow and will migrate the database
5. Once the migration is successful the user can happily use Airflow 2.0.
**Backport packages:**
We released packages that will allow us to use operators/hooks from Airflow 2.0 in Airflow 1.10.
Link: https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series
Status: **DONE**
**airflow upgrade-check**
To make it easier for users to upgrade from 1.10 to 2.0, we should create a single upgrade-check command in 1.10 that checks the common incompatibilities.
Link: https://github.com/apache/airflow/issues/8765
Status: **WIP**
**Migration guide:**
We have a lot of entries in `UPDATING.md`, but there is **no full procedure** that will answer the most important question - how to update Airflow to the latest version. Users expect step-by-step instructions from us that will explain what they should do. The guide should contain information about `airflow upgrade-check`, `airflow db upgrade`, backport package, etc.
More information: https://github.com/apache/airflow/issues/10089
Status: **WIP**
Additionally, we should make other changes to the documentation.
| Feature | Status | Issue |
|-|-|-|
| Prepare migration guide for CLI |WIP | https://github.com/apache/airflow/issues/10089 |
| Prepare migration guide for REST API | ✅ | https://github.com/apache/airflow/issues/8121 |
| Double check UPDATING.md for "2.0" doesn't include things already released | Not started | https://github.com/apache/airflow/issues/9641|
| Double check UPDATING.md for "2.0" doesn't include breaking changes in import paths | Not started | https://github.com/apache/airflow/issues/10104
**Depreciation warnings**
For each type of change, we should add notifications at the time of use so that the user doesn't have to read the documentation::
Name | Status | Issue | Description
-- | -- | -- | --
Python API | *WIP** | #8765 | Most of the changes have DeprecationWarning, but we can also detect common issues with upgrade-check.
CLI | WIP | #10109 | We can detect the use of the old command and propose to use the new one.
REST API | | #9888 | We send the message in HTTP headers.
Database schema | | #8765 | We can also detect known issues with upgrade-check.
Configuration | | #8765 | We can also detect known issues with upgrade-check.
# Task-force group
I heard that some members want to set up regular meetings to discuss the necessary work to be able to release Airflow 2.0.
More information: ?
Status: **WIP**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10152: Release Airflow 2.0
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10152:
URL: https://github.com/apache/airflow/issues/10152#issuecomment-668517490
Please do not comment on this ticket. To the discussion on the release of the Airflow version, I invite you to:
* Slack channel: [#development](https://apache-airflow.slack.com/archives/CCPRP7943) ([](https://s.apache.org/airflow-slack)) Slack channel
* [task-force meetings](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning).
* developer mailing list: [users-unsubscribe@airflow.apache.org](mailto:dev-subscribe@airflow.apache.org) ([Pony Mail](https://lists.apache.org/list.html?dev@airflow.apache.org))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #10152: Release Airflow 2.0
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #10152:
URL: https://github.com/apache/airflow/issues/10152#issuecomment-668517490
Please do not comment on this ticket. To the discussion on the release of the Airflow version, I invite you to:
* Slack channel: [#development](https://apache-airflow.slack.com/archives/CCPRP7943) ([](https://s.apache.org/airflow-slack))
* [task-force meetings](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning).
* developer mailing list: [users-unsubscribe@airflow.apache.org](mailto:dev-subscribe@airflow.apache.org) ([Pony Mail](https://lists.apache.org/list.html?dev@airflow.apache.org))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #10152: Release Airflow 2.0
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #10152:
URL: https://github.com/apache/airflow/issues/10152
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #10152: Release Airflow 2.0
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #10152:
URL: https://github.com/apache/airflow/issues/10152#issuecomment-668517490
Please do not comment on this ticket. To the discussion on the release of the Airflow version, I invite you to:
* Slack channel: [#development](https://apache-airflow.slack.com/archives/CCPRP7943) ([](https://s.apache.org/airflow-slack))
* [task-force meetings](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning).
* developer mailing list: [users-subscribe@airflow.apache.org](mailto:dev-subscribe@airflow.apache.org) ([Pony Mail](https://lists.apache.org/list.html?dev@airflow.apache.org))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10152: Release Airflow 2.0
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10152:
URL: https://github.com/apache/airflow/issues/10152#issuecomment-749130923
Task done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org