You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "prabhusneha (via GitHub)" <gi...@apache.org> on 2025/03/11 21:06:50 UTC
[PR] Setting Airflow context Environment variables for operators [airflow]
prabhusneha opened a new pull request, #47644:
URL: https://github.com/apache/airflow/pull/47644
Related: https://github.com/apache/airflow/issues/46418
This PR passes down the environment variables through context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2004820681
##########
airflow/models/taskinstance.py:
##########
@@ -124,6 +124,12 @@
from airflow.utils.task_instance_session import set_current_task_instance_session
from airflow.utils.timeout import timeout
from airflow.utils.xcom import XCOM_RETURN_KEY
+from airflow.version_compat import AIRFLOW_V_3_0_PLUS
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import context_to_airflow_vars
+else:
+ from airflow.utils.operator_helpers import context_to_airflow_vars # type: ignore[no-redef, attr-defined]
Review Comment:
For core (non providers), we do not need to do this exercise of v3 check as we can introduce breaking changes with 3.0, the issue is for providers as its a different package relying on core airflow. For those, we need those checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47644:
URL: https://github.com/apache/airflow/pull/47644#issuecomment-2743385341
This will require a rebase, let me take a stab at that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r1994636749
##########
providers/apache/hive/src/airflow/providers/apache/hive/operators/hive.py:
##########
@@ -26,8 +26,7 @@
from airflow.configuration import conf
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveCliHook
-from airflow.utils import operator_helpers
-from airflow.utils.operator_helpers import context_to_airflow_vars
+from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING, context_to_airflow_vars
Review Comment:
We'll need to import these from [common.compat provider](https://github.com/apache/airflow/blob/main/providers/common/compat/README.rst) for backward compat. or try maybe just use `AIRFLOW_V_3_0_PLUS` from `version_compat`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2006954721
##########
task-sdk/tests/task_sdk/execution_time/test_task_runner.py:
##########
@@ -916,6 +916,34 @@ def test_run_with_inlets_and_outlets(
mock_supervisor_comms.send_request.assert_any_call(msg=last_expected_msg, log=mock.ANY)
+@mock.patch("airflow.sdk.execution_time.task_runner.context_to_airflow_vars")
+@mock.patch.dict(os.environ, {}, clear=True)
+def test_execute_task_exports_env_vars(
+ mock_context_to_airflow_vars, create_runtime_ti, mock_supervisor_comms
+):
+ """Test that _execute_task exports airflow context to environment variables."""
+
+ def test_function():
+ return "test function"
+
+ task = PythonOperator(
+ task_id="test_task",
+ python_callable=test_function,
+ )
+
+ ti = create_runtime_ti(task=task, dag_id="dag_with_env_vars")
+ mock_supervisor_comms.get_message.return_value = OKResponse(
+ ok=True,
+ )
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r1992880393
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -75,6 +75,7 @@
set_current_context,
)
from airflow.utils.net import get_hostname
+from airflow.utils.operator_helpers import context_to_airflow_vars
Review Comment:
Moved `context_to_airflow_vars` to `airflow/sdk/execution_time/context.py` and fixed imports in all files.
Moved relevant tests from `airflow/tests/utils/test_operator_helpers.py` to `task-sdk/tests/task_sdk/execution_time/test_context.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2007812918
##########
providers/apache/hive/src/airflow/providers/apache/hive/version_compat.py:
##########
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Resolved.
##########
providers/apache/hive/src/airflow/providers/apache/hive/operators/hive.py:
##########
@@ -26,8 +26,15 @@
from airflow.configuration import conf
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveCliHook
-from airflow.utils import operator_helpers
-from airflow.utils.operator_helpers import context_to_airflow_vars
+from airflow.providers.common.compat.version_compat import AIRFLOW_V_3_0_PLUS
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING, context_to_airflow_vars
+else:
+ from airflow.utils.operator_helpers import ( # type: ignore[no-redef, attr-defined]
Review Comment:
Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on PR #47644:
URL: https://github.com/apache/airflow/pull/47644#issuecomment-2720142923
Added unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on PR #47644:
URL: https://github.com/apache/airflow/pull/47644#issuecomment-2720283347
Fixing the failing CI checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2007464750
##########
providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py:
##########
@@ -35,10 +35,17 @@
from airflow.configuration import conf
from airflow.exceptions import AirflowException
from airflow.hooks.base import BaseHook
+from airflow.providers.common.compat.version_compat import AIRFLOW_V_3_0_PLUS
from airflow.providers.common.sql.hooks.sql import DbApiHook
from airflow.security import utils
from airflow.utils.helpers import as_flattened_list
-from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING
+else:
+ from airflow.utils.operator_helpers import ( # type: ignore[no-redef, attr-defined]
+ AIRFLOW_VAR_NAME_FORMAT_MAPPING,
+ )
Review Comment:
Cool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2004820685
##########
airflow/version_compat.py:
##########
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
No need to have this file
##########
providers/apache/hive/src/airflow/providers/apache/hive/transfers/hive_to_mysql.py:
##########
@@ -26,7 +26,7 @@
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveServer2Hook
from airflow.providers.mysql.hooks.mysql import MySqlHook
-from airflow.utils.operator_helpers import context_to_airflow_vars
+from airflow.sdk.execution_time.context import context_to_airflow_vars
Review Comment:
Need version check here
##########
providers/apache/hive/tests/unit/apache/hive/transfers/test_hive_to_mysql.py:
##########
@@ -25,9 +25,15 @@
from airflow.providers.apache.hive.transfers.hive_to_mysql import HiveToMySqlOperator
from airflow.utils import timezone
-from airflow.utils.operator_helpers import context_to_airflow_vars
from unit.apache.hive import MockHiveServer2Hook, MockMySqlHook, TestHiveEnvironment
+from tests_common.test_utils.version_compat import AIRFLOW_V_3_0_PLUS
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import context_to_airflow_vars
+else:
+ from airflow.utils.operator_helpers import context_to_airflow_vars # type: ignore[no-redef, attr-defined]
Review Comment:
Yeah this is good
##########
task-sdk/src/airflow/sdk/execution_time/context.py:
##########
@@ -411,3 +448,70 @@ def context_update_for_unmapped(context: Context, task: BaseOperator) -> None:
context["params"] = process_params(
context["dag"], task, context["dag_run"].conf, suppress_exception=False
)
+
+
+def context_to_airflow_vars(context: Mapping[str, Any], in_env_var_format: bool = False) -> dict[str, str]:
+ """
+ Return values used to externally reconstruct relations between dags, dag_runs, tasks and task_instances.
+
+ Given a context, this function provides a dictionary of values that can be used to
+ externally reconstruct relations between dags, dag_runs, tasks and task_instances.
+ Default to abc.def.ghi format and can be made to ABC_DEF_GHI format if
+ in_env_var_format is set to True.
+
+ :param context: The context for the task_instance of interest.
+ :param in_env_var_format: If returned vars should be in ABC_DEF_GHI format.
+ :return: task_instance context as dict.
+ """
+ from airflow import settings
Review Comment:
```suggestion
from airflow import settings
from datetime import datetime
```
##########
task-sdk/src/airflow/sdk/execution_time/context.py:
##########
@@ -18,6 +18,7 @@
import contextlib
from collections.abc import Generator, Iterator, Mapping
+from datetime import datetime
Review Comment:
```suggestion
```
##########
providers/apache/hive/src/airflow/providers/apache/hive/version_compat.py:
##########
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
No need to introduce a new file since we now have compat provider to help us.
File: https://github.com/apache/airflow/blob/main/providers/common/compat/src/airflow/providers/common/compat/version_compat.py
And example usage: https://github.com/apache/airflow/blob/857d14b34d393000d5d23f4acc9264298688c6dd/providers/yandex/src/airflow/providers/yandex/links/yq.py#L31-L38
##########
providers/apache/hive/src/airflow/providers/apache/hive/transfers/hive_to_samba.py:
##########
@@ -26,7 +26,7 @@
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveServer2Hook
from airflow.providers.samba.hooks.samba import SambaHook
-from airflow.utils.operator_helpers import context_to_airflow_vars
+from airflow.sdk.execution_time.context import context_to_airflow_vars
Review Comment:
Need version check here
##########
task-sdk/tests/task_sdk/execution_time/test_task_runner.py:
##########
@@ -916,6 +916,34 @@ def test_run_with_inlets_and_outlets(
mock_supervisor_comms.send_request.assert_any_call(msg=last_expected_msg, log=mock.ANY)
+@mock.patch("airflow.sdk.execution_time.task_runner.context_to_airflow_vars")
+@mock.patch.dict(os.environ, {}, clear=True)
+def test_execute_task_exports_env_vars(
+ mock_context_to_airflow_vars, create_runtime_ti, mock_supervisor_comms
+):
+ """Test that _execute_task exports airflow context to environment variables."""
+
+ def test_function():
+ return "test function"
+
+ task = PythonOperator(
+ task_id="test_task",
+ python_callable=test_function,
+ )
+
+ ti = create_runtime_ti(task=task, dag_id="dag_with_env_vars")
+ mock_supervisor_comms.get_message.return_value = OKResponse(
+ ok=True,
+ )
Review Comment:
You shoudlnt' need this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r1992880640
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -599,6 +600,10 @@ def run(
error: BaseException | None = None
try:
context = ti.get_template_context()
+ # Export context to make it available for operators to use.
+ airflow_context_vars = context_to_airflow_vars(context, in_env_var_format=True)
+ os.environ.update(airflow_context_vars)
Review Comment:
Calling from `_execute_task` now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2001287010
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -711,6 +712,10 @@ def _execute_task(context: Context, ti: RuntimeTaskInstance):
# Populate the context var so ExecutorSafeguard doesn't complain
ctx.run(ExecutorSafeguard.tracker.set, task)
+ # Export context to make it available for operators to use.
Review Comment:
Modified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r1990717747
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -75,6 +75,7 @@
set_current_context,
)
from airflow.utils.net import get_hostname
+from airflow.utils.operator_helpers import context_to_airflow_vars
Review Comment:
Please don't import this from core to here.
Instead we need to move this helper fn in to task sdk somewhere
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -599,6 +600,10 @@ def run(
error: BaseException | None = None
try:
context = ti.get_template_context()
+ # Export context to make it available for operators to use.
+ airflow_context_vars = context_to_airflow_vars(context, in_env_var_format=True)
+ os.environ.update(airflow_context_vars)
Review Comment:
Let's call this inside _execute_task please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r1994636749
##########
providers/apache/hive/src/airflow/providers/apache/hive/operators/hive.py:
##########
@@ -26,8 +26,7 @@
from airflow.configuration import conf
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveCliHook
-from airflow.utils import operator_helpers
-from airflow.utils.operator_helpers import context_to_airflow_vars
+from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING, context_to_airflow_vars
Review Comment:
We'll need to import these from [common.compat provider](https://github.com/apache/airflow/blob/main/providers/common/compat/README.rst) for backward compat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47644:
URL: https://github.com/apache/airflow/pull/47644#issuecomment-2739183691
There's a static check failure too:
```
mypy providers/http/src providers/http/tests providers/sqlite/src providers/sqlite/tests providers/docker/src providers/docker/tests providers/jenkins/src providers/jenkins/tests providers/celery/src providers/celery/tests providers/postgres/src providers/postgres/tests providers/amazon/src providers/amazon/tests providers/opsgenie/src providers/opsgenie/tests providers/redis/src providers/redis/tests providers/smtp/src providers/smtp/tests providers/sftp/src providers/sftp/tests providers/samba/src providers/samba/tests providers/snowflake/src providers/snowflake/tests providers/alibaba/src providers/alibaba/tests providers/tableau/src providers/tableau/tests providers/slack/src providers/slack/tests providers/salesforce/src providers/salesforce/tests providers/yandex/src providers/yandex/tests providers/presto/src providers/presto/tests providers/vertica/src providers/vertica/tests providers/cncf/kubernetes/src providers/cncf/kubernetes/tests providers/opensearch/src providers/o
pensearch/tests providers/odbc/src providers/odbc/tests providers/openfaas/src providers/openfaas/tests providers/trino/src providers/trino/tests providers/arangodb/src providers/arangodb/tests providers/hashicorp/src providers/hashicorp/tests providers/mongo/src providers/mongo/tests providers/sendgrid/src providers/sendgrid/tests providers/papermill/src providers/papermill/tests providers/facebook/src providers/facebook/tests providers/standard/src providers/standard/tests providers/apprise/src providers/apprise/tests providers/pgvector/src providers/pgvector/tests providers/dingding/src providers/dingding/tests providers/mysql/src providers/mysql/tests providers/zendesk/src providers/zendesk/tests providers/google/src providers/google/tests providers/ssh/src providers/ssh/tests providers/exasol/src providers/exasol/tests providers/grpc/src providers/grpc/tests providers/influxdb/src providers/influxdb/tests providers/dbt/cloud/src providers/dbt/cloud/tests providers/segment/src p
roviders/segment/tests providers/cohere/src providers/cohere/tests providers/edge/src providers/edge/tests providers/cloudant/src providers/cloudant/tests providers/fab/src providers/fab/tests providers/pagerduty/src providers/pagerduty/tests providers/jdbc/src providers/jdbc/tests providers/telegram/src providers/telegram/tests providers/weaviate/src providers/weaviate/tests providers/ydb/src providers/ydb/tests providers/common/io/src providers/common/io/tests providers/common/compat/src providers/common/compat/tests providers/common/messaging/src providers/common/messaging/tests providers/common/sql/src providers/common/sql/tests providers/elasticsearch/src providers/elasticsearch/tests providers/asana/src providers/asana/tests providers/airbyte/src providers/airbyte/tests providers/ftp/src providers/ftp/tests providers/openlineage/src providers/openlineage/tests providers/github/src providers/github/tests providers/imap/src providers/imap/tests providers/neo4j/src providers/neo4
j/tests providers/discord/src providers/discord/tests providers/atlassian/jira/src providers/atlassian/jira/tests providers/apache/druid/src providers/apache/druid/tests providers/apache/drill/src providers/apache/drill/tests providers/apache/impala/src providers/apache/impala/tests providers/apache/pinot/src providers/apache/pinot/tests providers/apache/kylin/src providers/apache/kylin/tests providers/apache/spark/src providers/apache/spark/tests providers/apache/cassandra/src providers/apache/cassandra/tests providers/apache/iceberg/src providers/apache/iceberg/tests providers/apache/livy/src providers/apache/livy/tests providers/apache/hdfs/src providers/apache/hdfs/tests providers/apache/kafka/src providers/apache/kafka/tests providers/apache/hive/src providers/apache/hive/tests providers/apache/pig/src providers/apache/pig/tests providers/apache/beam/src providers/apache/beam/tests providers/apache/flink/src providers/apache/flink/tests providers/databricks/src providers/databr
icks/tests providers/datadog/src providers/datadog/tests providers/teradata/src providers/teradata/tests providers/openai/src providers/openai/tests providers/qdrant/src providers/qdrant/tests providers/oracle/src providers/oracle/tests providers/pinecone/src providers/pinecone/tests providers/microsoft/mssql/src providers/microsoft/mssql/tests providers/microsoft/winrm/src providers/microsoft/winrm/tests providers/microsoft/azure/src providers/microsoft/azure/tests providers/microsoft/psrp/src providers/microsoft/psrp/tests providers/singularity/src providers/singularity/tests --namespace-packages
providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py:46: error:
Module "airflow.utils.operator_helpers" has no attribute
"AIRFLOW_VAR_NAME_FORMAT_MAPPING" [attr-defined]
from airflow.utils.operator_helpers import (
^
providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py:46: error:
Name "AIRFLOW_VAR_NAME_FORMAT_MAPPING" already defined (possibly by an import)
[no-redef]
from airflow.utils.operator_helpers import (
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2006979425
##########
providers/apache/hive/src/airflow/providers/apache/hive/transfers/hive_to_mysql.py:
##########
@@ -25,8 +25,14 @@
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveServer2Hook
+from airflow.providers.common.compat.version_compat import AIRFLOW_V_3_0_PLUS
from airflow.providers.mysql.hooks.mysql import MySqlHook
-from airflow.utils.operator_helpers import context_to_airflow_vars
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import context_to_airflow_vars
+else:
+ from airflow.utils.operator_helpers import context_to_airflow_vars # type: ignore[no-redef, attr-defined]
Review Comment:
Same comment here as above
##########
providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py:
##########
@@ -35,10 +35,17 @@
from airflow.configuration import conf
from airflow.exceptions import AirflowException
from airflow.hooks.base import BaseHook
+from airflow.providers.common.compat.version_compat import AIRFLOW_V_3_0_PLUS
from airflow.providers.common.sql.hooks.sql import DbApiHook
from airflow.security import utils
from airflow.utils.helpers import as_flattened_list
-from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING
+else:
+ from airflow.utils.operator_helpers import ( # type: ignore[no-redef, attr-defined]
+ AIRFLOW_VAR_NAME_FORMAT_MAPPING,
+ )
Review Comment:
```suggestion
from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING # type: ignore[no-redef]
```
Only no-redef not enough?
##########
providers/standard/src/airflow/providers/standard/operators/bash.py:
##########
@@ -27,10 +27,15 @@
from airflow.exceptions import AirflowException, AirflowSkipException
from airflow.models.baseoperator import BaseOperator
from airflow.providers.standard.hooks.subprocess import SubprocessHook, SubprocessResult, working_directory
-from airflow.utils.operator_helpers import context_to_airflow_vars
+from airflow.providers.standard.version_compat import AIRFLOW_V_3_0_PLUS
from airflow.utils.session import NEW_SESSION, provide_session
from airflow.utils.types import ArgNotSet
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import context_to_airflow_vars
+else:
+ from airflow.utils.operator_helpers import context_to_airflow_vars # type: ignore[no-redef, attr-defined]
Review Comment:
Same comment as above
##########
providers/apache/hive/tests/unit/apache/hive/hooks/test_hive.py:
##########
@@ -30,9 +30,9 @@
from airflow.models.connection import Connection
from airflow.models.dag import DAG
from airflow.providers.apache.hive.hooks.hive import HiveCliHook, HiveMetastoreHook, HiveServer2Hook
+from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING
Review Comment:
Now this will need an AF3 check
##########
providers/apache/hive/src/airflow/providers/apache/hive/transfers/hive_to_samba.py:
##########
@@ -25,8 +25,13 @@
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveServer2Hook
+from airflow.providers.common.compat.version_compat import AIRFLOW_V_3_0_PLUS
from airflow.providers.samba.hooks.samba import SambaHook
-from airflow.utils.operator_helpers import context_to_airflow_vars
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import context_to_airflow_vars
+else:
+ from airflow.utils.operator_helpers import context_to_airflow_vars # type: ignore[no-redef, attr-defined]
Review Comment:
Same comment here as above
##########
generated/provider_dependencies.json:
##########
@@ -178,6 +178,7 @@
],
"cross-providers-deps": [
"amazon",
+ "common.compat",
Review Comment:
This will go away once you remove the AF3 check in core
##########
providers/trino/src/airflow/providers/trino/hooks/trino.py:
##########
@@ -32,7 +32,14 @@
from airflow.providers.common.sql.hooks.sql import DbApiHook
from airflow.providers.trino.version_compat import AIRFLOW_V_3_0_PLUS
from airflow.utils.helpers import exactly_one
-from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING, DEFAULT_FORMAT_PREFIX
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING, DEFAULT_FORMAT_PREFIX
+else:
+ from airflow.utils.operator_helpers import ( # type: ignore[no-redef, attr-defined]
Review Comment:
Same comment as above
##########
providers/apache/hive/src/airflow/providers/apache/hive/operators/hive.py:
##########
@@ -26,8 +26,15 @@
from airflow.configuration import conf
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveCliHook
-from airflow.utils import operator_helpers
-from airflow.utils.operator_helpers import context_to_airflow_vars
+from airflow.providers.common.compat.version_compat import AIRFLOW_V_3_0_PLUS
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING, context_to_airflow_vars
+else:
+ from airflow.utils.operator_helpers import ( # type: ignore[no-redef, attr-defined]
Review Comment:
Same comment here as above
##########
providers/presto/src/airflow/providers/presto/hooks/presto.py:
##########
@@ -30,7 +30,14 @@
from airflow.exceptions import AirflowException
from airflow.providers.common.sql.hooks.sql import DbApiHook
from airflow.providers.presto.version_compat import AIRFLOW_V_3_0_PLUS
-from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING, DEFAULT_FORMAT_PREFIX
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING, DEFAULT_FORMAT_PREFIX
+else:
+ from airflow.utils.operator_helpers import ( # type: ignore[no-redef, attr-defined]
Review Comment:
Same comment about only no-redef here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2007015664
##########
providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py:
##########
@@ -35,10 +35,17 @@
from airflow.configuration import conf
from airflow.exceptions import AirflowException
from airflow.hooks.base import BaseHook
+from airflow.providers.common.compat.version_compat import AIRFLOW_V_3_0_PLUS
from airflow.providers.common.sql.hooks.sql import DbApiHook
from airflow.security import utils
from airflow.utils.helpers import as_flattened_list
-from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING
+
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING
+else:
+ from airflow.utils.operator_helpers import ( # type: ignore[no-redef, attr-defined]
+ AIRFLOW_VAR_NAME_FORMAT_MAPPING,
+ )
Review Comment:
getting static check failures when only no-redef is used:
```
providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py:46: error:
Module "airflow.utils.operator_helpers" has no attribute
"AIRFLOW_VAR_NAME_FORMAT_MAPPING" [attr-defined]
from airflow.utils.operator_helpers import ( # type: ignore[no-re...
^
providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py:46: note: Error code "attr-defined" not covered by "type: ignore" comment
Found 6 errors in 5 files (checked 1178 source files)
Error 1 returned
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on PR #47644:
URL: https://github.com/apache/airflow/pull/47644#issuecomment-2742771879
@amoghrajesh Please take a look. CI checks are passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #47644:
URL: https://github.com/apache/airflow/pull/47644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r1991068003
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -75,6 +75,7 @@
set_current_context,
)
from airflow.utils.net import get_hostname
+from airflow.utils.operator_helpers import context_to_airflow_vars
Review Comment:
and then you can move over the tests too:
https://github.com/apache/airflow/blob/b85f14c4cf20cccd56aa06ce5ba4baf637a7f11c/tests/utils/test_operator_helpers.py#L53-L99
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r1992800672
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -75,6 +75,7 @@
set_current_context,
)
from airflow.utils.net import get_hostname
+from airflow.utils.operator_helpers import context_to_airflow_vars
Review Comment:
We have `get_template_context` in task runner itself, so maybe you can move that helper in here somewhere or even consider the `airflow/sdk/execution_time/context.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r1994772666
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -711,6 +712,10 @@ def _execute_task(context: Context, ti: RuntimeTaskInstance):
# Populate the context var so ExecutorSafeguard doesn't complain
ctx.run(ExecutorSafeguard.tracker.set, task)
+ # Export context to make it available for operators to use.
Review Comment:
```suggestion
# Export context in os.environ to make it available for operators to use.
```
##########
providers/apache/hive/src/airflow/providers/apache/hive/operators/hive.py:
##########
@@ -26,8 +26,7 @@
from airflow.configuration import conf
from airflow.models import BaseOperator
from airflow.providers.apache.hive.hooks.hive import HiveCliHook
-from airflow.utils import operator_helpers
-from airflow.utils.operator_helpers import context_to_airflow_vars
+from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING, context_to_airflow_vars
Review Comment:
We can just use the v3 check too
##########
task-sdk/tests/task_sdk/execution_time/test_task_runner.py:
##########
@@ -890,6 +890,34 @@ def test_run_with_inlets_and_outlets(
mock_supervisor_comms.send_request.assert_any_call(msg=last_expected_msg, log=mock.ANY)
+@mock.patch("airflow.sdk.execution_time.task_runner.context_to_airflow_vars")
+@mock.patch.dict(os.environ, {}, clear=True)
+def test_execute_task_exports_env_vars(
+ mock_context_to_airflow_vars, create_runtime_ti, mock_supervisor_comms
+):
+ """Test that _execute_task exports airflow context to environment variables."""
+
+ def test_function():
+ return "test function"
+
+ task = PythonOperator(
+ task_id="test_task",
+ python_callable=test_function,
+ )
+
+ ti = create_runtime_ti(task=task, dag_id="dag_with_env_vars")
+ mock_supervisor_comms.get_message.return_value = OKResponse(
+ ok=True,
+ )
Review Comment:
I dont think this is the right thing?
##########
providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py:
##########
@@ -36,9 +36,9 @@
from airflow.exceptions import AirflowException
from airflow.hooks.base import BaseHook
from airflow.providers.common.sql.hooks.sql import DbApiHook
+from airflow.sdk.execution_time.context import AIRFLOW_VAR_NAME_FORMAT_MAPPING
Review Comment:
When we move around things into the sdk, one thing to note is the back-compat.
To do that, we usually do a version check and import for the right version, one such example is here
https://github.com/apache/airflow/pull/47008/files#diff-90e854af5044e923499f865ed449cfacc40ca2db5afc47ac0cca278a29fcc9bfR47-R52
This will have to be done for all providers
##########
task-sdk/src/airflow/sdk/execution_time/context.py:
##########
@@ -411,3 +448,70 @@ def context_update_for_unmapped(context: Context, task: BaseOperator) -> None:
context["params"] = process_params(
context["dag"], task, context["dag_run"].conf, suppress_exception=False
)
+
+
+def context_to_airflow_vars(context: Mapping[str, Any], in_env_var_format: bool = False) -> dict[str, str]:
Review Comment:
Type of context is probably: `context: Context` now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on PR #47644:
URL: https://github.com/apache/airflow/pull/47644#issuecomment-2716461004
Can we add a test for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47644:
URL: https://github.com/apache/airflow/pull/47644#issuecomment-2716516781
@prabhusneha can we please add some unit tests and also test with the dag that was originally provided in the bug report?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "atul-astronomer (via GitHub)" <gi...@apache.org>.
atul-astronomer commented on PR #47644:
URL: https://github.com/apache/airflow/pull/47644#issuecomment-2747912603
As per the description, now env variables are not printed in the logs explicitly as it was in AF2 but if user wants, they can print it using 'print(os.environ.get('AIRFLOW_CTX_DAG_OWNER'))'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Setting Airflow context Environment variables for operators [airflow]
Posted by "prabhusneha (via GitHub)" <gi...@apache.org>.
prabhusneha commented on code in PR #47644:
URL: https://github.com/apache/airflow/pull/47644#discussion_r2006955081
##########
task-sdk/tests/task_sdk/execution_time/test_task_runner.py:
##########
@@ -890,6 +890,34 @@ def test_run_with_inlets_and_outlets(
mock_supervisor_comms.send_request.assert_any_call(msg=last_expected_msg, log=mock.ANY)
+@mock.patch("airflow.sdk.execution_time.task_runner.context_to_airflow_vars")
+@mock.patch.dict(os.environ, {}, clear=True)
+def test_execute_task_exports_env_vars(
+ mock_context_to_airflow_vars, create_runtime_ti, mock_supervisor_comms
+):
+ """Test that _execute_task exports airflow context to environment variables."""
+
+ def test_function():
+ return "test function"
+
+ task = PythonOperator(
+ task_id="test_task",
+ python_callable=test_function,
+ )
+
+ ti = create_runtime_ti(task=task, dag_id="dag_with_env_vars")
+ mock_supervisor_comms.get_message.return_value = OKResponse(
+ ok=True,
+ )
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org