You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@apache.org> on 2021/03/08 21:41:18 UTC
[Meeting Notes] Airflow Dev Call - 3 March 2021
Hi all,
Here are the notes from our Airflow dev call last week. Thank you to all
who joined the call.
Here is a quick summary of the call.
*Summary Notes*:
- *Add a policy document around all our Policies / rules in the Github
Repo itself*
- This should cover any decision we take as PMC / Committers, for
example:
- Versioning Policy - Core, Providers and all others
- Committers have a binding vote on AIP Releases
- Lazy Consensus thread will be created on the mailing list for each
decision agreed on the Dev Call
- *Providers Release Policy*
- *Batch vs Ad-hoc*:
- Release Manager would default to release Providers in Batch
- ad-hoc releases are OK (i.e. if there is a critical bug that
needs fixing in a single provider)
- *Frequency:*
- For Batch release, we will release *every month *(starting of
the month - 1 to 7 most likely)
- Just a note that it generally takes around a week for the vote
to pass even though we have 72 hours minimum period
- *Testing*
- License and Signature Checks are mandatory (following the ASF
rules)
- For Providers, not all changes require strict testing -- you
make a judgement based on the changes for a particular provider
- For Airflow core it is different, smoke tests should be carried
out and we rely on community testing too (Kaxil to create a separate
Mailing list thread to discuss this further since there wasn't 100%
agreement)
- *Doc-only changes*
- When we have doc-only changes for Providers (during
batch-release), we should still release a new version.
Majority on the Dev
call agreed that releasing docs asap is good instead of waiting for
the next release with a code-change.
- *Feedback on AIPs*
- AIP-39 Richer scheduler_interval
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-39+Richer+scheduler_interval>
- Foreign keys between TaskInstance and DagRun will be changed
from execution_date and dag_id to dagrun_id.
- This and any other changes with columns will need migration that
might be slow for deployments with many Dags and large TIs or
DagRuns in
Metadata DB.
- Kevin Yang from Airbnb said that they have tried something
similar but instead of dagrun_id they used DagRun.id
(auto-incrementing
id for each DagRun) which had a performance impact.
- AIP-38 Modern Web Application
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application>
is
already being Voted on: link
<https://lists.apache.org/thread.html/r646c453087fa982e7277823d7b638a238314adbcb361fa93ac1230f6%40%3Cdev.airflow.apache.org%3E>
- *TaskGroups support in Tree View*
- We agreed that this would be good to have since we are planning to
remove SubDags for TaskGroups
- A separate Mailing list thread has been created for more details
and further discussion: link
<https://lists.apache.org/thread.html/r17626aff377aa4914edb8ed3ab9ce17193a8d1615e7388c77db76b4d%40%3Cdev.airflow.apache.org%3E>
Notes are also posted on Confluence too:
*Doc Link*:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Calls
Best regards,
Kaxil
Re: [Meeting Notes] Airflow Dev Call - 3 March 2021
Posted by Tomasz Urbaszek <tu...@apache.org>.
Thanks Kaxil for the notes!
Tomek
On Mon, 8 Mar 2021 at 22:41, Kaxil Naik <ka...@apache.org> wrote:
>
> Hi all,
>
> Here are the notes from our Airflow dev call last week. Thank you to all who joined the call.
> Here is a quick summary of the call.
>
> Summary Notes:
>
> Add a policy document around all our Policies / rules in the Github Repo itself
>
> This should cover any decision we take as PMC / Committers, for example:
>
> Versioning Policy - Core, Providers and all others
> Committers have a binding vote on AIP Releases
>
> Lazy Consensus thread will be created on the mailing list for each decision agreed on the Dev Call
>
> Providers Release Policy
>
> Batch vs Ad-hoc:
>
> Release Manager would default to release Providers in Batch
> ad-hoc releases are OK (i.e. if there is a critical bug that needs fixing in a single provider)
>
> Frequency:
>
> For Batch release, we will release every month (starting of the month - 1 to 7 most likely)
> Just a note that it generally takes around a week for the vote to pass even though we have 72 hours minimum period
>
> Testing
>
> License and Signature Checks are mandatory (following the ASF rules)
> For Providers, not all changes require strict testing -- you make a judgement based on the changes for a particular provider
> For Airflow core it is different, smoke tests should be carried out and we rely on community testing too (Kaxil to create a separate Mailing list thread to discuss this further since there wasn't 100% agreement)
>
> Doc-only changes
>
> When we have doc-only changes for Providers (during batch-release), we should still release a new version. Majority on the Dev call agreed that releasing docs asap is good instead of waiting for the next release with a code-change.
>
> Feedback on AIPs
>
> AIP-39 Richer scheduler_interval
>
> Foreign keys between TaskInstance and DagRun will be changed from execution_date and dag_id to dagrun_id.
> This and any other changes with columns will need migration that might be slow for deployments with many Dags and large TIs or DagRuns in Metadata DB.
> Kevin Yang from Airbnb said that they have tried something similar but instead of dagrun_id they used DagRun.id (auto-incrementing id for each DagRun) which had a performance impact.
>
> AIP-38 Modern Web Application is already being Voted on: link
>
> TaskGroups support in Tree View
>
> We agreed that this would be good to have since we are planning to remove SubDags for TaskGroups
> A separate Mailing list thread has been created for more details and further discussion: link
>
>
> Notes are also posted on Confluence too:
> Doc Link: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Calls
>
> Best regards,
> Kaxil