You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kacpermuda (via GitHub)" <gi...@apache.org> on 2025/03/07 14:39:13 UTC
[PR] feat: Adjust DBT OpenLineage to Airflow 3 and improve logging [airflow]
kacpermuda opened a new pull request, #47500:
URL: https://github.com/apache/airflow/pull/47500
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Main change in this PR is adjusting the way `_get_logical_date` function works, now it should mimic OpenLineage listener's behavior. This is crucial, as OL run_id generated here must match the one generated by the listener, so that the parent-child relationship is correctly created.
Additionally, some logs have been added that will be useful for debugging any issues with dbt OL extraction.
cc @mobuchowski
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Adjust DBT OpenLineage to Airflow 3 and improve logging [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #47500:
URL: https://github.com/apache/airflow/pull/47500#discussion_r1987110829
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py:
##########
@@ -110,8 +114,15 @@ def generate_openlineage_events_from_dbt_cloud_run(
# catalog is available only if docs are generated
catalog = None
- with suppress(Exception):
+ try:
+ log.debug("Retrieving information about catalog artifact from DBT.")
catalog = operator.hook.get_job_run_artifact(operator.run_id, path="catalog.json").json()["data"]
+ except Exception: # type: ignore
+ log.info(
+ "Openlineage could not find DBT catalog artifact, usually available when docs are generated."
+ "Proceeding with metadata extraction. "
+ "If you see error logs above about `HTTP error: Not Found` it's safe to ignore them."
+ )
Review Comment:
Makes sense. I have no better idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Adjust DBT OpenLineage to Airflow 3 and improve logging [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #47500:
URL: https://github.com/apache/airflow/pull/47500#discussion_r1987051276
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -286,6 +286,10 @@ def get_openlineage_facets_on_complete(self, task_instance) -> OperatorLineage:
if isinstance(self.run_id, int) and self.wait_for_termination is True:
return generate_openlineage_events_from_dbt_cloud_run(operator=self, task_instance=task_instance)
+ self.log.info(
+ "Extraction of OpenLineage events from DBT will be skipped: "
+ "`run_id` is not set OR `self.wait_for_termination` is False. "
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Adjust DBT OpenLineage to Airflow 3 and improve logging [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #47500:
URL: https://github.com/apache/airflow/pull/47500#discussion_r1985315425
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -286,6 +286,10 @@ def get_openlineage_facets_on_complete(self, task_instance) -> OperatorLineage:
if isinstance(self.run_id, int) and self.wait_for_termination is True:
return generate_openlineage_events_from_dbt_cloud_run(operator=self, task_instance=task_instance)
+ self.log.info(
+ "Extraction of OpenLineage events from DBT will be skipped: "
+ "`run_id` is not set OR `self.wait_for_termination` is False. "
Review Comment:
If you're logging this, you might as well display which one of those is true
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py:
##########
@@ -110,8 +114,15 @@ def generate_openlineage_events_from_dbt_cloud_run(
# catalog is available only if docs are generated
catalog = None
- with suppress(Exception):
+ try:
+ log.debug("Retrieving information about catalog artifact from DBT.")
catalog = operator.hook.get_job_run_artifact(operator.run_id, path="catalog.json").json()["data"]
+ except Exception: # type: ignore
+ log.info(
+ "Openlineage could not find DBT catalog artifact, usually available when docs are generated."
+ "Proceeding with metadata extraction. "
+ "If you see error logs above about `HTTP error: Not Found` it's safe to ignore them."
+ )
Review Comment:
If that's true and the feature is optional, we should not log those errors and maybe display one collective error message at the end if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Adjust DBT OpenLineage to Airflow 3 and improve logging [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #47500:
URL: https://github.com/apache/airflow/pull/47500#discussion_r1987043289
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py:
##########
@@ -110,8 +114,15 @@ def generate_openlineage_events_from_dbt_cloud_run(
# catalog is available only if docs are generated
catalog = None
- with suppress(Exception):
+ try:
+ log.debug("Retrieving information about catalog artifact from DBT.")
catalog = operator.hook.get_job_run_artifact(operator.run_id, path="catalog.json").json()["data"]
+ except Exception: # type: ignore
+ log.info(
+ "Openlineage could not find DBT catalog artifact, usually available when docs are generated."
+ "Proceeding with metadata extraction. "
+ "If you see error logs above about `HTTP error: Not Found` it's safe to ignore them."
+ )
Review Comment:
These logs come from HTTPHook, that is used underneath. There is no way to pass some arguments that deep so that It does not log this error, so my only idea was to write some kind of logs interceptor to be used for the duration of this call. It looked to complex compared to the benefits, so I decided against. Do you have some simpler idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Adjust DBT OpenLineage to Airflow 3 and improve logging [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski merged PR #47500:
URL: https://github.com/apache/airflow/pull/47500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Adjust DBT OpenLineage to Airflow 3 and improve logging [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #47500:
URL: https://github.com/apache/airflow/pull/47500#discussion_r1985175589
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py:
##########
@@ -17,39 +17,41 @@
from __future__ import annotations
import asyncio
+import logging
import re
-from contextlib import suppress
from typing import TYPE_CHECKING
-from packaging.version import parse
-
-from airflow import __version__ as airflow_version
+from airflow.providers.dbt.cloud.version_compat import AIRFLOW_V_2_10_PLUS, AIRFLOW_V_3_0_PLUS
if TYPE_CHECKING:
- from packaging.version import Version
-
from airflow.models.taskinstance import TaskInstance
from airflow.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperator
from airflow.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensor
from airflow.providers.openlineage.extractors.base import OperatorLineage
-_AIRFLOW_VERSION: Version = parse(parse(airflow_version).base_version)
+log = logging.getLogger(__name__)
def _get_logical_date(task_instance):
# todo: remove when min airflow version >= 3.0
- if parse("3") > _AIRFLOW_VERSION:
- return task_instance.execution_date
- return task_instance.logical_date
+ if AIRFLOW_V_3_0_PLUS:
+ dagrun = task_instance.get_template_context()["dag_run"]
+ return dagrun.logical_date or dagrun.run_after
Review Comment:
@mobuchowski Does this look right? I was trying to mimic OL's listener behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Adjust DBT OpenLineage to Airflow 3 and improve logging [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #47500:
URL: https://github.com/apache/airflow/pull/47500#discussion_r1985314038
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py:
##########
@@ -17,39 +17,41 @@
from __future__ import annotations
import asyncio
+import logging
import re
-from contextlib import suppress
from typing import TYPE_CHECKING
-from packaging.version import parse
-
-from airflow import __version__ as airflow_version
+from airflow.providers.dbt.cloud.version_compat import AIRFLOW_V_2_10_PLUS, AIRFLOW_V_3_0_PLUS
if TYPE_CHECKING:
- from packaging.version import Version
-
from airflow.models.taskinstance import TaskInstance
from airflow.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperator
from airflow.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensor
from airflow.providers.openlineage.extractors.base import OperatorLineage
-_AIRFLOW_VERSION: Version = parse(parse(airflow_version).base_version)
+log = logging.getLogger(__name__)
def _get_logical_date(task_instance):
# todo: remove when min airflow version >= 3.0
- if parse("3") > _AIRFLOW_VERSION:
- return task_instance.execution_date
- return task_instance.logical_date
+ if AIRFLOW_V_3_0_PLUS:
+ dagrun = task_instance.get_template_context()["dag_run"]
+ return dagrun.logical_date or dagrun.run_after
Review Comment:
Yes, given that you don't have to deal with tests without `dag_run` :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org