You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Komalpawar22 (via GitHub)" <gi...@apache.org> on 2025/07/16 10:56:17 UTC

[D] Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API [airflow]

GitHub user Komalpawar22 created a discussion: Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API

# Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API

**Repository**: `apache/airflow`  
**Type**: Discussion (Q&A section)  
**Submit to**: https://github.com/apache/airflow/discussions  
**Category**: Q&A  
**Tags**: `airflow-3.0`, `traditional-dag`, `operators`, `taskflow-api`, `migration`

---

## Summary

I'm just simply seeking official clarification on Airflow 3.0 support for the **complete traditional DAG writing approach** using DAG context and operators exclusively, without any TaskFlow API decorators.

## Current Setup & Approach

**Airflow Version**: 2.10.4 (Kubernetes deployment)  
**Executor**: KubernetesExecutor  
**Deployment**: GKE with Helm Chart 1.14.0

We exclusively use the traditional approach for both DAG and task definitions:

```python
# Our current pattern - NO decorators anywhere
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
from airflow.operators.kubernetes import KubernetesOperator
from datetime import datetime

def my_python_function():
    return "Processing data..."

with DAG(
    'traditional_dag_example',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily',
    catchup=False
) as dag:
    
    python_task = PythonOperator(
        task_id='data_processing',
        python_callable=my_python_function
    )
    
    bash_task = BashOperator(
        task_id='bash_command',
        bash_command='echo "Traditional approach works!"'
    )
    
    k8s_task = KubernetesOperator(
        task_id='kubernetes_job',
        # ... kubernetes config
    )
    
    python_task >> bash_task >> k8s_task
```

**We are NOT using TaskFlow API:**
```python
# We avoid this approach completely
@dag(start_date=datetime(2023, 1, 1))
def taskflow_example():
    @task
    def my_task():
        return "Hello World"
    
    my_task()
```

## Specific Questions for Airflow 3.0

### 1. **Complete Traditional Support**
Will Airflow 3.0 continue to fully support the traditional DAG context + operators approach without requiring ANY `@task` or `@dag` decorators?

### 2. **Operator Ecosystem Compatibility**
Will all existing operators (PythonOperator, BashOperator, KubernetesOperator, custom operators, etc.) continue to work seamlessly within DAG context in Airflow 3.0?

### 3. **Migration Requirements**
For large deployments with 100+ traditional DAGs using only operators and DAG context, is there any **forced migration requirement** to TaskFlow API in 3.0?

### 4. **Feature Parity & Performance**
- Will the traditional approach have full feature parity with TaskFlow API in 3.0?
- Are there any performance implications or feature limitations?
- Will new Airflow 3.0 features be available to traditional DAG patterns?

### 5. **Long-term Support & Documentation**
- What's the long-term commitment for supporting operator-based DAG writing?
- Will documentation continue to cover traditional patterns alongside TaskFlow API?

## Business Context

**Scale**: 100+ production DAGs using traditional approach  
**Pattern**: Heavy use of PythonOperator, BashOperator, KubernetesOperator  
**Dependencies**: Complex task relationships using `>>` operators  
**Codebase**: Zero decorators - all task logic in separate functions passed to operators  
**Infrastructure**: Established CI/CD pipelines built around traditional patterns  

## What We Need

Official confirmation from maintainers on:

1. ✅ **Zero migration requirement** for traditional DAG + operator approach in Airflow 3.0
2. ✅ **Full feature parity** between traditional and TaskFlow approaches  
3. ✅ **Long-term support commitment** for operator-based DAG writing
4. ✅ **Documentation continuity** for traditional patterns

We specifically want to continue with pure **operator + DAG context patterns** and avoid any decorator-based approach.

## Additional Context

- **Custom Docker Image**: Using custom Airflow image with additional dependencies
- **Git-based DAG Deployment**: DAGs are synced from Git repository
- **Kubernetes-native**: Running on GKE with KubernetesExecutor
- **Production Workload**: Mission-critical data processing pipelines
- **Team Expertise**: Development team trained on traditional operator patterns

---

**Thank you for clarifying the future direction of traditional DAG patterns in Airflow 3.0!**

GitHub link: https://github.com/apache/airflow/discussions/53413

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API

Unless it's stated as deprecated, there are no plans to remove feature from Airflow. This might change in the future though - Airflow is a living thing and the community developing it might make different decisions in the future (but as one of the maintaiers I have not heard of any plans of removing traditional operators).

Note, that this is not "official" confirmation you are seeking for. I am just one of the maintainers and the views are my own. I doubt you can get any "official" confirmation from anyone about any "future" events. But - until now at least you can use our SemVer intention of not removing features from Airflow and apply it to whatever features you use - and make your decisions based on that.

Airflow is licenced with Apache Software Foundation licence that explicitly states that you have no guarantees whatsoever:

https://www.apache.org/licenses/LICENSE-2.0

> 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.

So there is no "official" guarantees you can get from anyone here - there are only intentions expressed by maintainers in README and you are solely responsible to assume any risks connected with lack of such "official statements" and "guarantees". Which makes sense because you pay exactly 0 USD to the maintainers, so you are getting exact equivalent of this in terms of guarantees (i.e. none).

GitHub link: https://github.com/apache/airflow/discussions/53413#discussioncomment-13783846

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API

Note, that this is not "official" confirmation you are seeking for. I am just one of the mainers and the views are my own. I doubt you can get any "official" confirmation from anyone about any "future" events. But - until now at least you can use our SemVer intention of not removing features from Airflow and apply it to whatever features you use - and make your decisions based on that.

Airflow is licenced with Apache Software Foundation licence that explicitly states that you have no guarantees whatsoever:

GitHub link: https://github.com/apache/airflow/discussions/53413#discussioncomment-13783846

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Airflow 3.0: Future of Traditional DAG Context + Operators Approach vs TaskFlow API

Airflow is licenced with Apache Software Foundation licence that explicitly states that you have no guarantees whatsoever:

GitHub link: https://github.com/apache/airflow/discussions/53413#discussioncomment-13783846

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org