You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@apache.org> on 2020/11/30 14:33:09 UTC
[Reminder] Airflow 2.0 Dev call Today in 3 hours & Meeting Notes for
Last call
Hi all,
I would like to remind whoever is interested in helping in Airflow 2.0
Release process that we will have our weekly developer call today:
*Airflow 2.0 Dev Call Calendar Link*:
https://calendar.google.com/calendar/embed?src=c_dhdh3bdjc42c4ngtnpg7ovcs9g%40group.calendar.google.com&ctz=Europe%2FLondon
*Time*: 5:30-6:30 PM GMT
*Zoom Link*:
https://astronomer.zoom.us/j/93263978170?pwd=Ym9wMUJUNkc5cnJ0VnNabVRtaVBvZz09
Some useful links:
- Planning Doc & Meeting Notes:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
- 2.0 Milestone: https://github.com/apache/airflow/milestone/3
- Issue Tracker: https://github.com/apache/airflow/issues/10152
Last Meeting call was short, the meeting notes for which is published at
https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#14:23Nov2020.
Airflow 1.10.13 and Upgrade-Check 1.0.0 were released on 25 Nov.
1.10.14 should hopefully be released this week, that fixes a bug in 1.10.13
Regards,
Kaxil
Re: [Reminder] Airflow 2.0 Dev call Today in 3 hours & Meeting Notes
for Last call
Posted by Kaxil Naik <ka...@gmail.com>.
Sure, will add that
On Mon, Nov 30, 2020 at 2:53 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I wanted to add one topic to the discussion:
>
> How viable it is to expect migration from 1.10 DB to 2.0 will working and
> whether we should recommend as a community to start Airflow Metadata from
> scratch, vs. supporting 1.10.13 -> 2.0 DB migration as a fully supported
> option.
>
> We've been discussing it in the past meeting but I do not think we reached
> (and did not capture) any finl conclusion (and we might have different
> assumptions here).
>
> The https://github.com/apache/airflow/blob/master/UPGRADING_TO_2.0.md has
> one mention of "airflow db upgrade" ("As part of running db upgrade,
> existing permissions will be migrated for you."). But the DB migration step
> is not included in the document and I do not know of any efforts of testing
> a large-scale migration of existing Airflow 1.10 DB to 2.0.
>
> I know from past experiences that depending on databases and the amount of
> data in the DB such migrations can take hours or in extreme cases even days
> (depending on how optimized the migration scripts are).
>
> I think we need to discuss and capture at least community recommendations
> on that and make it clear in the migration docs.
>
> J.
>
>
> On Mon, Nov 30, 2020 at 3:33 PM Kaxil Naik <ka...@apache.org> wrote:
>
>> Hi all,
>>
>> I would like to remind whoever is interested in helping in Airflow 2.0
>> Release process that we will have our weekly developer call today:
>>
>> *Airflow 2.0 Dev Call Calendar Link*:
>> https://calendar.google.com/calendar/embed?src=c_dhdh3bdjc42c4ngtnpg7ovcs9g%40group.calendar.google.com&ctz=Europe%2FLondon
>> *Time*: 5:30-6:30 PM GMT
>> *Zoom Link*:
>> https://astronomer.zoom.us/j/93263978170?pwd=Ym9wMUJUNkc5cnJ0VnNabVRtaVBvZz09
>>
>>
>> Some useful links:
>>
>> - Planning Doc & Meeting Notes:
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
>> - 2.0 Milestone: https://github.com/apache/airflow/milestone/3
>> - Issue Tracker: https://github.com/apache/airflow/issues/10152
>>
>>
>> Last Meeting call was short, the meeting notes for which is published at
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#14:23Nov2020
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%2314:23Nov2020>.
>> Airflow 1.10.13 and Upgrade-Check 1.0.0 were released on 25 Nov.
>>
>> 1.10.14 should hopefully be released this week, that fixes a bug in
>> 1.10.13
>>
>> Regards,
>> Kaxil
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: [Reminder] Airflow 2.0 Dev call Today in 3 hours & Meeting Notes
for Last call
Posted by Jarek Potiuk <Ja...@polidea.com>.
I wanted to add one topic to the discussion:
How viable it is to expect migration from 1.10 DB to 2.0 will working and
whether we should recommend as a community to start Airflow Metadata from
scratch, vs. supporting 1.10.13 -> 2.0 DB migration as a fully supported
option.
We've been discussing it in the past meeting but I do not think we reached
(and did not capture) any finl conclusion (and we might have different
assumptions here).
The https://github.com/apache/airflow/blob/master/UPGRADING_TO_2.0.md has
one mention of "airflow db upgrade" ("As part of running db upgrade,
existing permissions will be migrated for you."). But the DB migration step
is not included in the document and I do not know of any efforts of testing
a large-scale migration of existing Airflow 1.10 DB to 2.0.
I know from past experiences that depending on databases and the amount of
data in the DB such migrations can take hours or in extreme cases even days
(depending on how optimized the migration scripts are).
I think we need to discuss and capture at least community recommendations
on that and make it clear in the migration docs.
J.
On Mon, Nov 30, 2020 at 3:33 PM Kaxil Naik <ka...@apache.org> wrote:
> Hi all,
>
> I would like to remind whoever is interested in helping in Airflow 2.0
> Release process that we will have our weekly developer call today:
>
> *Airflow 2.0 Dev Call Calendar Link*:
> https://calendar.google.com/calendar/embed?src=c_dhdh3bdjc42c4ngtnpg7ovcs9g%40group.calendar.google.com&ctz=Europe%2FLondon
> *Time*: 5:30-6:30 PM GMT
> *Zoom Link*:
> https://astronomer.zoom.us/j/93263978170?pwd=Ym9wMUJUNkc5cnJ0VnNabVRtaVBvZz09
>
>
> Some useful links:
>
> - Planning Doc & Meeting Notes:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
> - 2.0 Milestone: https://github.com/apache/airflow/milestone/3
> - Issue Tracker: https://github.com/apache/airflow/issues/10152
>
>
> Last Meeting call was short, the meeting notes for which is published at
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#14:23Nov2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%2314:23Nov2020>.
> Airflow 1.10.13 and Upgrade-Check 1.0.0 were released on 25 Nov.
>
> 1.10.14 should hopefully be released this week, that fixes a bug in 1.10.13
>
> Regards,
> Kaxil
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>