You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mobuchowski (via GitHub)" <gi...@apache.org> on 2026/04/07 14:50:21 UTC
[PR] Mobuchowski/cache connection [airflow]
mobuchowski opened a new pull request, #64843:
URL: https://github.com/apache/airflow/pull/64843
`emit_lineage_from_sql_extras` called `hook.get_connection()` twice per SQL extra — once in `_resolve_namespace` and once inside `get_openlineage_facets_with_sql`. For N extras from the same hook (common when a task runs multiple SQL statements) this is N×2 redundant round-trips that all return the same connection object. Each call hits SecretsManager (miss) then the Airflow API server, making it the dominant cost of SQL hook lineage processing.
Fix: build a `conn_id → hook` mapping before the loop, then define three `@functools.cache`-decorated local closures keyed by `conn_id`:
- `_get_connection` — one `hook.get_connection()` call per unique `conn_id`
- `_get_database_info` — derived from `_get_connection`, cached separately
- `_get_namespace` — derived from `_get_database_info`, cached separately
Each concern is cached independently via `functools.cache`; no manual dict, no tuple packing. `hook.get_connection()` fires exactly once per unique `conn_id` regardless of how many SQL extras share it.
Also includes two related fixes:
- `extractors/manager.py`: wrap `get_hook_lineage()` in try/except so an exception in SQL extras processing can't silently suppress the task-level COMPLETE event via `@print_warning`
- `plugins/listener.py`: call `logging.shutdown()` before `os._exit(0)` in the fork child so buffered log messages (including failure warnings) are flushed before exit
##### Was generative AI tooling used to co-author this PR?
- [ ] Yes (please specify the tool below)
Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cache connections in OpenLineage SQL hook lineage [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #64843:
URL: https://github.com/apache/airflow/pull/64843#discussion_r3052727359
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -790,8 +790,21 @@ def _fork_execute(self, callable, callable_name: str):
if not AIRFLOW_V_3_0_PLUS:
configure_orm(disable_connection_pool=True)
self.log.debug("Executing OpenLineage process - %s - pid %s", callable_name, os.getpid())
- callable()
- self.log.debug("Process with current pid finishes after %s", callable_name)
+ try:
+ callable()
+ self.log.debug("Process with current pid finishes after %s", callable_name)
+ except Exception:
+ self.log.warning(
+ "OpenLineage %s process failed. This has no impact on actual task execution status.",
+ callable_name,
+ exc_info=True,
+ )
+ finally:
+ # os._exit(0) bypasses Python's atexit/stdio flush. Explicitly shut down
+ # logging so buffered records (including any warnings above) are flushed
+ # before the process exits. Without this, the final log lines are silently
+ # dropped, making failures invisible.
+ logging.shutdown()
Review Comment:
Yeah it does not affect main process from what I can see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cache connections in OpenLineage SQL hook lineage [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #64843:
URL: https://github.com/apache/airflow/pull/64843#discussion_r3051231911
##########
providers/openlineage/src/airflow/providers/openlineage/extractors/manager.py:
##########
@@ -142,9 +142,26 @@ def extract_metadata(
task_info,
)
self.log.debug("OpenLineage extraction failure details:", exc_info=True)
- elif (hook_lineage := self.get_hook_lineage(task_instance, task_instance_state)) is not None:
- return hook_lineage
else:
+ # No extractor found — fall back to hook lineage. This call must be wrapped in
+ # try/except: it runs emit_lineage_from_sql_extras → _create_ol_event_pair which
+ # is not guarded internally. An uncaught exception here would propagate into
+ # on_success()'s @print_warning, silently suppressing the task-level COMPLETE event.
Review Comment:
Fixed the comment/log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cache connections in OpenLineage SQL hook lineage [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #64843:
URL: https://github.com/apache/airflow/pull/64843#discussion_r3045965439
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -790,8 +790,21 @@ def _fork_execute(self, callable, callable_name: str):
if not AIRFLOW_V_3_0_PLUS:
configure_orm(disable_connection_pool=True)
self.log.debug("Executing OpenLineage process - %s - pid %s", callable_name, os.getpid())
- callable()
- self.log.debug("Process with current pid finishes after %s", callable_name)
+ try:
+ callable()
+ self.log.debug("Process with current pid finishes after %s", callable_name)
+ except Exception:
+ self.log.warning(
+ "OpenLineage %s process failed. This has no impact on actual task execution status.",
+ callable_name,
+ exc_info=True,
+ )
+ finally:
+ # os._exit(0) bypasses Python's atexit/stdio flush. Explicitly shut down
+ # logging so buffered records (including any warnings above) are flushed
+ # before the process exits. Without this, the final log lines are silently
+ # dropped, making failures invisible.
+ logging.shutdown()
Review Comment:
I am not sure if there are any implications of doing that in a provider, for other logs flowing in airflow. Can you double check that it's isolated here?
##########
providers/openlineage/src/airflow/providers/openlineage/utils/sql_hook_lineage.py:
##########
@@ -70,6 +71,27 @@ def emit_lineage_from_sql_extras(task_instance, sql_extras: list, is_successful:
events: list[RunEvent] = []
query_count = 0
+ # Build conn_id -> hook mapping before iterating. Hook instances are not hashable so
+ # conn_id (a plain string) is used as the @cache key throughout.
+ _hook_by_conn_id = {_get_hook_conn_id(e.context): e.context for e in sql_extras}
Review Comment:
Is it enough to reduce hooks to using conn_id as key here? Maybe id(hook)? Wondering if it's possible for two hooks to send HLL, with the same db connection, but different params resulting in different database info ?
##########
providers/openlineage/src/airflow/providers/openlineage/extractors/manager.py:
##########
@@ -142,9 +142,26 @@ def extract_metadata(
task_info,
)
self.log.debug("OpenLineage extraction failure details:", exc_info=True)
- elif (hook_lineage := self.get_hook_lineage(task_instance, task_instance_state)) is not None:
- return hook_lineage
else:
+ # No extractor found — fall back to hook lineage. This call must be wrapped in
+ # try/except: it runs emit_lineage_from_sql_extras → _create_ol_event_pair which
+ # is not guarded internally. An uncaught exception here would propagate into
+ # on_success()'s @print_warning, silently suppressing the task-level COMPLETE event.
Review Comment:
This path is not only for COMPLETE events, so we should adjust comments and logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cache connections in OpenLineage SQL hook lineage [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski merged PR #64843:
URL: https://github.com/apache/airflow/pull/64843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cache connections in OpenLineage SQL hook lineage [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #64843:
URL: https://github.com/apache/airflow/pull/64843#discussion_r3066476391
##########
providers/openlineage/src/airflow/providers/openlineage/sqlparser.py:
##########
@@ -473,13 +473,24 @@ def _get_tables_hierarchy(
def get_openlineage_facets_with_sql(
- hook: DbApiHook, sql: str | list[str], conn_id: str, database: str | None, use_connection: bool = True
+ hook: DbApiHook,
+ sql: str | list[str],
+ conn_id: str,
+ database: str | None,
+ use_connection: bool = True,
+ connection=None,
+ database_info=None,
) -> OperatorLineage | None:
Review Comment:
The new `connection`/`database_info` parameters are untyped and positional-capable, which makes incorrect positional calls easier and reduces readability for API consumers. Consider making these keyword-only (e.g., `*, connection=..., database_info=...`) and adding concrete types (even if broad, e.g., `connection: Any | None`, `database_info: Any | None`) to clarify intent and improve static analysis.
##########
providers/openlineage/src/airflow/providers/openlineage/sqlparser.py:
##########
@@ -473,13 +473,24 @@ def _get_tables_hierarchy(
def get_openlineage_facets_with_sql(
- hook: DbApiHook, sql: str | list[str], conn_id: str, database: str | None, use_connection: bool = True
+ hook: DbApiHook,
+ sql: str | list[str],
+ conn_id: str,
+ database: str | None,
+ use_connection: bool = True,
+ connection=None,
+ database_info=None,
) -> OperatorLineage | None:
- connection = hook.get_connection(conn_id)
- try:
- database_info = hook.get_openlineage_database_info(connection)
- except AttributeError:
- database_info = None
+ # Accept pre-fetched connection and database_info to avoid redundant hook.get_connection()
+ # calls when processing multiple SQL extras from the same hook. Each get_connection() call
+ # hits SecretsManager (miss) then the Airflow API server — passing these in caches that cost.
+ if connection is None or database_info is None:
+ connection = hook.get_connection(conn_id)
+ if database_info is None:
+ try:
+ database_info = hook.get_openlineage_database_info(connection)
+ except AttributeError:
+ database_info = None
if database_info is None:
Review Comment:
When a caller passes `connection` but not `database_info`, this branch still re-fetches the connection and overwrites the provided one. This defeats the purpose of accepting a pre-fetched connection and can reintroduce the expensive `get_connection()` call. Split the logic so `hook.get_connection(conn_id)` is only called when `connection is None`, and only fetch `database_info` when it is missing (using the existing `connection`).
```suggestion
if connection is None:
connection = hook.get_connection(conn_id)
if database_info is None:
try:
database_info = hook.get_openlineage_database_info(connection)
except AttributeError:
database_info = None
if database_info is None:
```
##########
providers/openlineage/src/airflow/providers/openlineage/sqlparser.py:
##########
@@ -473,13 +473,24 @@ def _get_tables_hierarchy(
def get_openlineage_facets_with_sql(
- hook: DbApiHook, sql: str | list[str], conn_id: str, database: str | None, use_connection: bool = True
+ hook: DbApiHook,
+ sql: str | list[str],
+ conn_id: str,
+ database: str | None,
+ use_connection: bool = True,
+ connection=None,
+ database_info=None,
) -> OperatorLineage | None:
- connection = hook.get_connection(conn_id)
- try:
- database_info = hook.get_openlineage_database_info(connection)
- except AttributeError:
- database_info = None
+ # Accept pre-fetched connection and database_info to avoid redundant hook.get_connection()
+ # calls when processing multiple SQL extras from the same hook. Each get_connection() call
+ # hits SecretsManager (miss) then the Airflow API server — passing these in caches that cost.
Review Comment:
The last sentence is grammatically unclear ('passing these in caches that cost'). Consider rewording to something like: '... — passing these in avoids that cost.' This comment explains an important performance motivation, so clarity helps future maintainers.
```suggestion
# hits SecretsManager (miss) then the Airflow API server — passing these in avoids that cost.
```
##########
providers/openlineage/src/airflow/providers/openlineage/utils/sql_hook_lineage.py:
##########
@@ -70,6 +71,34 @@ def emit_lineage_from_sql_extras(task_instance, sql_extras: list, is_successful:
events: list[RunEvent] = []
query_count = 0
+ # Build hook identity -> (hook, conn_id) mapping before iterating.
+ # Using id(hook) as cache key instead of conn_id ensures distinct hook instances
+ # with the same conn_id but different params are cached separately.
+ _hook_info: dict[int, tuple[object, str | None]] = {}
Review Comment:
The PR description says the caches are keyed by `conn_id` via a `conn_id → hook` mapping, but the implementation explicitly keys caches by `id(hook)` and caches separately per hook instance. Either update the PR description to match the implemented behavior (and rationale), or switch the implementation to `conn_id` keying if that was the original intent.
##########
providers/openlineage/src/airflow/providers/openlineage/sqlparser.py:
##########
@@ -473,13 +473,24 @@ def _get_tables_hierarchy(
def get_openlineage_facets_with_sql(
- hook: DbApiHook, sql: str | list[str], conn_id: str, database: str | None, use_connection: bool = True
+ hook: DbApiHook,
+ sql: str | list[str],
+ conn_id: str,
+ database: str | None,
+ use_connection: bool = True,
+ connection=None,
+ database_info=None,
) -> OperatorLineage | None:
- connection = hook.get_connection(conn_id)
- try:
- database_info = hook.get_openlineage_database_info(connection)
- except AttributeError:
- database_info = None
+ # Accept pre-fetched connection and database_info to avoid redundant hook.get_connection()
+ # calls when processing multiple SQL extras from the same hook. Each get_connection() call
+ # hits SecretsManager (miss) then the Airflow API server — passing these in caches that cost.
+ if connection is None or database_info is None:
+ connection = hook.get_connection(conn_id)
+ if database_info is None:
+ try:
+ database_info = hook.get_openlineage_database_info(connection)
+ except AttributeError:
+ database_info = None
Review Comment:
The new `connection`/`database_info` override behavior isn’t directly covered by unit tests. Add tests that assert: (1) when `connection` is provided and `database_info` is missing, `hook.get_connection` is not called, and (2) when `database_info` is provided and `connection` is missing, `get_connection` is called exactly once and `get_openlineage_database_info` is not called.
```suggestion
if connection is None:
connection = hook.get_connection(conn_id)
if database_info is None:
try:
database_info = hook.get_openlineage_database_info(connection)
except AttributeError:
database_info = None
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cache connections in OpenLineage SQL hook lineage [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #64843:
URL: https://github.com/apache/airflow/pull/64843#discussion_r3045987628
##########
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py:
##########
@@ -790,8 +790,21 @@ def _fork_execute(self, callable, callable_name: str):
if not AIRFLOW_V_3_0_PLUS:
configure_orm(disable_connection_pool=True)
self.log.debug("Executing OpenLineage process - %s - pid %s", callable_name, os.getpid())
- callable()
- self.log.debug("Process with current pid finishes after %s", callable_name)
+ try:
+ callable()
+ self.log.debug("Process with current pid finishes after %s", callable_name)
+ except Exception:
+ self.log.warning(
+ "OpenLineage %s process failed. This has no impact on actual task execution status.",
+ callable_name,
+ exc_info=True,
+ )
+ finally:
+ # os._exit(0) bypasses Python's atexit/stdio flush. Explicitly shut down
+ # logging so buffered records (including any warnings above) are flushed
+ # before the process exits. Without this, the final log lines are silently
+ # dropped, making failures invisible.
+ logging.shutdown()
Review Comment:
We're on our own process there, but will doublecheck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cache connections in OpenLineage SQL hook lineage [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #64843:
URL: https://github.com/apache/airflow/pull/64843#discussion_r3056392651
##########
providers/openlineage/src/airflow/providers/openlineage/utils/sql_hook_lineage.py:
##########
@@ -70,6 +71,27 @@ def emit_lineage_from_sql_extras(task_instance, sql_extras: list, is_successful:
events: list[RunEvent] = []
query_count = 0
+ # Build conn_id -> hook mapping before iterating. Hook instances are not hashable so
+ # conn_id (a plain string) is used as the @cache key throughout.
+ _hook_by_conn_id = {_get_hook_conn_id(e.context): e.context for e in sql_extras}
Review Comment:
That makes sense, even if it feels like super rare situation, it won't break the most common use case. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] cache connections in OpenLineage SQL hook lineage [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #64843:
URL: https://github.com/apache/airflow/pull/64843#issuecomment-4231713589
Tests are failing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org