You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Daniel Standish <dp...@gmail.com> on 2024/12/20 23:25:45 UTC
[DISCUSS] removal of uniqueness in airflow and partition-driven dag style
OK folks, let's get into it.
AIP-83 fundamentally alters Airflow's dag run semantics in a way that I
think no one fully appreciated when it was proposed and adopted.
Previously the logical key of dag_run was dag_id + logical_date. That
combination defined what a dag run means, and uniquely identified a dag
run. In AIP-83, We removed the constraint, which is easy enough; but we
did not do anything to maintain backcompat, or to address the semantic
ambiguities introduced. Ultimately, we need to decide as a community what
to do.
I don't expect to impose my will upon the community. I let go of outcome.
But I aim to help folks understand the issue so that we can collectively
arrive at a good decision for the project. And I would only ask that, if
you intend to engage and comment, that you try to be patient and read and
consider the whole doc. It probably won't take as long as it looks.
One fundamental question is, should we continue to support the old
semantics or not. Do users expect or depend on the old semantics? And do
we care?
If not, then we ought to be clear about this with users. If we want to
support the old semantics, then we need to decide what that looks like.
Here is a draft AIP amendment
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-83+amendment+to+support+classic+Airflow+authoring+style>
where I've outlined the the history, motivations, concerns, and options
that I am aware of, and as I understand them. There are 5 of them. My
vote would be anything between 1 and 3, with mild preference for 3.
Personally, I don't see any reason why we cannot or should not allow users
to elect to design dags with the old semantics. To be clear, I am no
champion of this design pattern; in my former life as data engineer, I
would say my dags were execution-date-driven something less than 1% of the
time. It makes a lot of sense for a hive / presto / athena shop, but much
less e.g. for snowflake. But I recognize that, many folks do use it, and
Airflow has gotten this far assuming *all* dags are designed this way, so,
maybe we should allow users to optionally keep those semantics in Airflow 3.
The other thing that we should think through is, *why* are we removing
uniqueness. Why do we care? What does this buy us? If it's just about
allowing manual triggering of dags that don't care about logical date, then
one option would be to just make logical date nullable and call it a day.
Alright, happy holidays. Let's try and step our way towards consensus and
figure out the best path forward for Airflow.
Re: [DISCUSS] removal of uniqueness in airflow and partition-driven dag style
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
Alright, I have tried to synthesize the feedback on the Option 2
clarification document
<https://cwiki.apache.org/confluence/display/AIRFLOW/Option+2+clarification+doc+WIP>
.
The proposal that I plan to bring for a vote will be the two sections,
Uncontroversial
elements
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337676722#Option2clarificationdocWIP-Uncontroversialelements>,
and Resolution of questions
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337676722#Option2clarificationdocWIP-Resolutionofquestions>
.
I'll leave this for another day before calling for a vote, to allow folks
to review my synthesis of the comments in the resolution section, in case I
got something wrong.
Thanks
On Tue, Jan 7, 2025 at 8:13 AM Daniel Standish <
daniel.standish@astronomer.io> wrote:
> OK just an update.
>
> Recall the document outlining options was created AIP-83 amendment to
> support classic Airflow authoring style
> <https://cwiki.apache.org/confluence/x/Ngv0Ew>.
>
> From discussions on that page, it seemed that some variant of option 2 has
> the strongest support.
>
> Accordingly, I have created a document, Option 2 clarification doc
> <https://cwiki.apache.org/confluence/x/sokgF>, to outline the points of
> agreement and unresolved questions, specifically focusing on option 2. The
> goal is to resolve each of the remaining questions, update this document to
> reflect the consensus proposal, and then bring it to the list for a vote.
>
> If you are interested or concerned with the outcome, you are welcome to
> engage on that doc.
>
>
>
>
>
>
Re: [DISCUSS] removal of uniqueness in airflow and partition-driven dag style
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
OK just an update.
Recall the document outlining options was created AIP-83 amendment to
support classic Airflow authoring style
<https://cwiki.apache.org/confluence/x/Ngv0Ew>.
From discussions on that page, it seemed that some variant of option 2 has
the strongest support.
Accordingly, I have created a document, Option 2 clarification doc
<https://cwiki.apache.org/confluence/x/sokgF>, to outline the points of
agreement and unresolved questions, specifically focusing on option 2. The
goal is to resolve each of the remaining questions, update this document to
reflect the consensus proposal, and then bring it to the list for a vote.
If you are interested or concerned with the outcome, you are welcome to
engage on that doc.
Re: [DISCUSS] removal of uniqueness in airflow and partition-driven dag style
Posted by Jarek Potiuk <ja...@potiuk.com>.
Commented on the page. Avi - you might not be aware that the discussion in
the doc progresses quickly :) - and there are already some changes and even
Daniel changed his preferences (and added more options) - so likely it's
best to comment there.
On Sun, Dec 22, 2024 at 3:57 PM Abhishek Bhakat
<ab...@astronomer.io.invalid> wrote:
> Can we do a hybrid of Option 2 and 3?
>
> class DAG:
> def __init__(
> self,
> dag_id: str,
> partition: Optional[str] = "auto", # "auto", "strict", "none"
> logical_dates: bool = True, # For backward compatibility
> ):
>
> 1. partition_mode="auto": Backward compat, logs warnings when multiple runs
> with same logical_date.
> 2. partition_mode="strict": Can be our traditional method. One run per
> logical_date.
> 3. partition_mode="none": logical_date becomes optional.
>
> For Migrations:
>
> # Legacy DAG - automatically uses strict mode or auto mode with
> warnings (*can decide on it later*)
> dag = DAG(
> 'legacy_dag',
> schedule_interval='0 0 * * *')
> # Modern DAG - explicit non-partitioned approach
> dag = DAG(
> 'modern_dag',
> schedule_interval='0 0 * * *',
> partition='none',
> logical_dates=False)
>
> Thanks,
> Avi
>
> On Sat, Dec 21, 2024 at 1:07 AM Mehta, Shubham <sh...@amazon.com.invalid>
> wrote:
>
> > Daniel, thank you for bringing this up.
> > This is definitely an important discussion, and I encourage others,
> > especially Airflow users, to share their thoughts as well. I’ve also
> shared
> > my opinion on the wiki.
> >
> > Shubham
> >
> >
>
Re: [DISCUSS] removal of uniqueness in airflow and partition-driven dag style
Posted by Abhishek Bhakat <ab...@astronomer.io.INVALID>.
Can we do a hybrid of Option 2 and 3?
class DAG:
def __init__(
self,
dag_id: str,
partition: Optional[str] = "auto", # "auto", "strict", "none"
logical_dates: bool = True, # For backward compatibility
):
1. partition_mode="auto": Backward compat, logs warnings when multiple runs
with same logical_date.
2. partition_mode="strict": Can be our traditional method. One run per
logical_date.
3. partition_mode="none": logical_date becomes optional.
For Migrations:
# Legacy DAG - automatically uses strict mode or auto mode with
warnings (*can decide on it later*)
dag = DAG(
'legacy_dag',
schedule_interval='0 0 * * *')
# Modern DAG - explicit non-partitioned approach
dag = DAG(
'modern_dag',
schedule_interval='0 0 * * *',
partition='none',
logical_dates=False)
Thanks,
Avi
On Sat, Dec 21, 2024 at 1:07 AM Mehta, Shubham <sh...@amazon.com.invalid>
wrote:
> Daniel, thank you for bringing this up.
> This is definitely an important discussion, and I encourage others,
> especially Airflow users, to share their thoughts as well. I’ve also shared
> my opinion on the wiki.
>
> Shubham
>
>
Re: [DISCUSS] removal of uniqueness in airflow and partition-driven dag style
Posted by "Mehta, Shubham" <sh...@amazon.com.INVALID>.
Daniel, thank you for bringing this up.
This is definitely an important discussion, and I encourage others, especially Airflow users, to share their thoughts as well. I’ve also shared my opinion on the wiki.
Shubham
Re: [DISCUSS] removal of uniqueness in airflow and partition-driven dag style
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
Daniel,
Thank you for writing this up. I commented on the wiki page.
Vikram
On Fri, Dec 20, 2024 at 3:26 PM Daniel Standish <dp...@gmail.com>
wrote:
> OK folks, let's get into it.
>
> AIP-83 fundamentally alters Airflow's dag run semantics in a way that I
> think no one fully appreciated when it was proposed and adopted.
> Previously the logical key of dag_run was dag_id + logical_date. That
> combination defined what a dag run means, and uniquely identified a dag
> run. In AIP-83, We removed the constraint, which is easy enough; but we
> did not do anything to maintain backcompat, or to address the semantic
> ambiguities introduced. Ultimately, we need to decide as a community what
> to do.
>
> I don't expect to impose my will upon the community. I let go of outcome.
> But I aim to help folks understand the issue so that we can collectively
> arrive at a good decision for the project. And I would only ask that, if
> you intend to engage and comment, that you try to be patient and read and
> consider the whole doc. It probably won't take as long as it looks.
>
> One fundamental question is, should we continue to support the old
> semantics or not. Do users expect or depend on the old semantics? And do
> we care?
>
> If not, then we ought to be clear about this with users. If we want to
> support the old semantics, then we need to decide what that looks like.
>
> Here is a draft AIP amendment
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-83+amendment+to+support+classic+Airflow+authoring+style
> >
> where I've outlined the the history, motivations, concerns, and options
> that I am aware of, and as I understand them. There are 5 of them. My
> vote would be anything between 1 and 3, with mild preference for 3.
>
> Personally, I don't see any reason why we cannot or should not allow users
> to elect to design dags with the old semantics. To be clear, I am no
> champion of this design pattern; in my former life as data engineer, I
> would say my dags were execution-date-driven something less than 1% of the
> time. It makes a lot of sense for a hive / presto / athena shop, but much
> less e.g. for snowflake. But I recognize that, many folks do use it, and
> Airflow has gotten this far assuming *all* dags are designed this way, so,
> maybe we should allow users to optionally keep those semantics in Airflow
> 3.
>
> The other thing that we should think through is, *why* are we removing
> uniqueness. Why do we care? What does this buy us? If it's just about
> allowing manual triggering of dags that don't care about logical date, then
> one option would be to just make logical date nullable and call it a day.
>
> Alright, happy holidays. Let's try and step our way towards consensus and
> figure out the best path forward for Airflow.
>