You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dheerajturaga (via GitHub)" <gi...@apache.org> on 2025/10/29 00:10:33 UTC
[PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
dheerajturaga opened a new pull request, #57459:
URL: https://github.com/apache/airflow/pull/57459
The dag_processor was unable to retrieve connections from the database,
causing GitHook (and other hooks) to fail with:
AirflowNotFoundException: The conn_id `<conn_id>` isn't defined
Root cause: dag_processor was running in FALLBACK context, which only
loads EnvironmentVariablesBackend, not MetastoreBackend. This meant
connections stored in the database were inaccessible.
The dag_processor is a server-side component that needs database access
for connection retrieval, similar to the scheduler and API server.
Fix: Set _AIRFLOW_PROCESS_CONTEXT=server in DagProcessorJobRunner._execute()
to enable MetastoreBackend, matching the pattern already used in:
- SchedulerJobRunner (scheduler_job_runner.py:1064)
- API FastAPI server (api_fastapi/main.py:24)
This ensures the dag_processor uses the correct secrets backend chain:
EnvironmentVariablesBackend → MetastoreBackend (database access)
Affects: DAG bundle processing with GitHook and any other hooks that
rely on database-stored connections during DAG parsing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2471424691
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
I am surprised how did it work for 3.1.0 -- then? Coz the server/client `_AIRFLOW_PROCESS_CONTEXT` env var was only added in 3.1.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2471419723
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
>The dag_processor is a server-side component that needs database access
for connection retrieval, similar to the scheduler and API server.
That isn't true. Dag Processor is a client side component -- since it runs user code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2474282757
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
@kaxil even though dag-processor is "client-side":
1. Only the child parser subprocess has SUPERVISOR_COMMS (is truly client-side)
2. The parent manager runs as a Job with database access (looks server-side)
I believe the fix I made (_AIRFLOW_PROCESS_CONTEXT=server) is correct because:
- Manager is server-side infrastructure
- It has database access via Job
- It's analogous to Scheduler (which sets server)
- GitHook initialization happens in manager, not child
What do you think? the git dag bundles feature is broken out of the box without this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2471423576
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
`DagFileProcessorManager` for now does have DB access but that will go away soon.
Could you provide the stacktrace for me to look closely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix connection retrieval in `DagProcessorManager` for bundle initialization [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #57459:
URL: https://github.com/apache/airflow/pull/57459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2471607873
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
@kaxil, this is the error I see in 3.1.0
```
Could not create GitHook bundle_name=airflow_dags version=None bare_repo_path=PosixPath('/tmp/airflow/dag_bundles/airflow_dags/bare')
repo_path=PosixPath('/tmp/airflow/dag_bundles/airflow_dags/tracking_repo') versions_path=PosixPath('/tmp/airflow/dag_bundles/airflow_dags/versions') git_conn_id=airflow_dags conn_id=airflow_dags exc=ImportError("cannot import name 'SUPERVISOR_COMMS'
from 'airflow.sdk.execution_time.task_runner' (/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py)")
```
This is the error I see in 3.1.1
```
Traceback (most recent call last):
File "/files/dags/test_connection_production.py", line 145, in test_connection_retrieval
connection = GitHook.get_connection(conn_id)
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/hook.py", line 61, in get_connection
conn = Connection.get(conn_id)
File "/opt/airflow/task-sdk/src/airflow/sdk/definitions/connection.py", line 225, in get
return _get_connection(conn_id)
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/context.py", line 175, in _get_connection
raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `airflow_dags` isn't defined
```
This broke in 3.1.0 but I bypassed the issue by specifying the repo_url through `dag_bundle_config_list`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2479545820
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
The problem with this env var is also that the child processes will inherit the env var.
While not a problem in itself, since `if hasattr(task_runner, "SUPERVISOR_COMMS") and task_runner.SUPERVISOR_COMMS is not None:` is checked first, but for correctness it shouldn't be there.
Let me take a stab at it with a PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2479545820
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
The problem with this env var is also that the child processes will inherit the env var.
While not a problem in itself, since `if hasattr(task_runner, "SUPERVISOR_COMMS") and task_runner.SUPERVISOR_COMMS is not None:` is checked first, but for correctness it shouldn't be there.
Let me take a stab at pushing changes to this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix connection retrieval in `DagProcessorManager` for bundle initialization [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on PR #57459:
URL: https://github.com/apache/airflow/pull/57459#issuecomment-3470682338
Thanks @kaxil !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2471422147
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
And the client context does have access to `ExecutionAPISecretsBackend`
https://github.com/apache/airflow/blob/a6d3e2d00639f95e619647329668f62a68b7f1f2/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1825-L1833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2476180453
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
>The parent manager runs as a Job with database access (looks server-side)
That's only temporary though until [AIP-92](https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services).
If we do keep this as solution, we should add a TODO that this is a temp solution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2479608140
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
Pushed, 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57459:
URL: https://github.com/apache/airflow/pull/57459#discussion_r2476180453
##########
airflow-core/src/airflow/jobs/dag_processor_job_runner.py:
##########
@@ -56,6 +57,9 @@ def __init__(
)
def _execute(self) -> int | None:
+ # Mark this as a server context for secrets backend detection
+ os.environ["_AIRFLOW_PROCESS_CONTEXT"] = "server"
+
Review Comment:
>The parent manager runs as a Job with database access (looks server-side)
That's only temporary though until [AIP-92](https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services).
If we do keep this as solution, we should add a TODO that this is a temp solution and we do proper test for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix GitHook connection retrieval in dag_processor by setting server context [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on PR #57459:
URL: https://github.com/apache/airflow/pull/57459#issuecomment-3459136246
cc: @kaxil this is a critical bug and we should aim for 3.1.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org