[D] XCom Limitations and Immutable Operators in Airflow 3.x [airflow]

["merkushov (via GitHub)" <gi...@apache.org>]

GitHub user merkushov created a discussion: XCom Limitations and Immutable Operators in Airflow 3.x

I’m running into an issue with XCom and data passing between tasks.

As part of the migration from Airflow 2.x to 3.x, I’ve noticed that accessing context.task_instance no longer works as it used to — it seems that the context is now immutable.

In my case, I need to pass an array from one task and combine it with some static data in the next operator. Here’s a simplified example of what I’m trying to achieve:

```python
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
from airflow.providers.http.operators.http import HttpOperator

def return_list():
    return [{"accounts":["apple", "banana", "cherry"]}]

def enrich_context(task):
    def _enricher(context) -> None:
        account_ids = []
        for item in context.get("task_instance").xcom_pull(task_ids=task.task_id):
            account_ids.extend(item.get("accounts", []))

        context.get("task").data["ids"] = account_ids
        return None

    return _enricher

with DAG(
    dag_id="DEBUG_XCOM",
) as dag:
    payload_task = PythonOperator(
        task_id="payload",
        python_callable=return_list,
    )

    http_task = HttpOperator(
        task_id="http_request",
        endpoint="/log-request",
        method="POST",
        data={
            "foo": 1,
        },
        headers={"Content-Type": "application/json"},
        pre_execute=enrich_context(payload_task),
    )

    payload_task >> http_task
```

Are there any recommended alternatives for passing data between tasks in Airflow 3.x without relying on TaskFlow decorators?

GitHub link: https://github.com/apache/airflow/discussions/54772

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] XCom Limitations and Immutable Operators in Airflow 3.x [airflow]

["merkushov (via GitHub)" <gi...@apache.org>]

GitHub user merkushov added a comment to the discussion: XCom Limitations and Immutable Operators in Airflow 3.x

@potiuk you suggest almost the same thing, but override the main method (`execute`) instead of using `pre_execute`. This doesn't work because I can't modify any Task object (Operator) via Context anymore

GitHub link: https://github.com/apache/airflow/discussions/54772#discussioncomment-14180766

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] XCom Limitations and Immutable Operators in Airflow 3.x [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk edited a comment on the discussion: XCom Limitations and Immutable Operators in Airflow 3.x

You can extend HttpOperator:

```python
class MyEnrichedHttpOperator(HttpOperator):

    def execute(context):
          enrich_context_and_store_stuff_in_objedt(self).
          super().execute(context) # your example is not clear ho wthe task/ids are going to be used in HttpOperator so likely you need to modify a bit more. Not sure if setting stuff in self is enought - but it should be.
```

and you can use such operator. The method to enrich things might be even shared between different operators of yours.

GitHub link: https://github.com/apache/airflow/discussions/54772#discussioncomment-14179136

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] XCom Limitations and Immutable Operators in Airflow 3.x [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: XCom Limitations and Immutable Operators in Airflow 3.x

You can extend HttpOperator:

```python
class MyEnrichedHttpOperator(HttpOperator):

    def execute(context):
          enrich_context_and_store_stuff_in_objedt(self).
          super.execut(context) # your example is not clear ho wthe task/ids are going to be used in HttpOperator so likely you need to modify a bit more. Not sure if setting stuff in self is enought - but it should be.
```

and you can use such operator. The method to enrich things might be even shared between different operators of yours.

GitHub link: https://github.com/apache/airflow/discussions/54772#discussioncomment-14179136

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] XCom Limitations and Immutable Operators in Airflow 3.x [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: XCom Limitations and Immutable Operators in Airflow 3.x

I am not suggesting modifying context. I am suggesting you to override the operator with "Your operator" that will store the required data inside the operator class not the context

As you noticed - context is immutable and you should not modify it. Full stop. 

My suggestion is to modify the fields of the operator and modify it's behaviour based on those.

GitHub link: https://github.com/apache/airflow/discussions/54772#discussioncomment-14190620

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org