You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2024/05/26 17:51:37 UTC
[I] OpenLineage 1.15.0 breaks dbt openlineage integration when running on top of Airlfow 2.9.1 [airflow]
potiuk opened a new issue, #39854:
URL: https://github.com/apache/airflow/issues/39854
### Body
The lates openlineage libraries (1.15.0) break dbt integration when running tests in Airflow 2.9.1 compatibility tests.
Example failure: https://github.com/apache/airflow/actions/runs/9244532008/job/25430068337#step:11:3221
```python
______ TestGenerateOpenLineageEventsFromDbtCloudRun.test_generate_events _______
self = <tests.providers.dbt.cloud.utils.test_openlineage.TestGenerateOpenLineageEventsFromDbtCloudRun object at 0x7f564e38e550>
mock_get_job_run_artifact = <MagicMock name='get_job_run_artifact' id='140008094655296'>
mock_get_project = <MagicMock name='get_project' id='140008093522048'>
mock_get_job_run = <MagicMock name='get_job_run' id='140008093512800'>
mock_build_task_instance_run_id = <MagicMock name='build_task_instance_run_id' id='140008092178608'>
mock_get_openlineage_listener = <MagicMock name='get_openlineage_listener' id='140008092297824'>
@patch("airflow.providers.openlineage.plugins.listener.get_openlineage_listener")
@patch("airflow.providers.openlineage.plugins.adapter.OpenLineageAdapter.build_task_instance_run_id")
@patch.object(DbtCloudHook, "get_job_run")
@patch.object(DbtCloudHook, "get_project")
@patch.object(DbtCloudHook, "get_job_run_artifact")
def test_generate_events(
self,
mock_get_job_run_artifact,
mock_get_project,
mock_get_job_run,
mock_build_task_instance_run_id,
mock_get_openlineage_listener,
):
mock_operator = MagicMock(spec=DbtCloudRunJobOperator)
mock_operator.account_id = None
mock_hook = DbtCloudHook()
mock_operator.hook = mock_hook
mock_get_job_run.return_value.json.return_value = read_file_json(
"tests/providers/dbt/cloud/test_data/job_run.json"
)
mock_get_project.return_value.json.return_value = {
"data": {
"connection": {
"type": "snowflake",
"details": {
"account": "gp21411.us-east-1",
"database": "SANDBOX",
"warehouse": "HUMANS",
"allow_sso": False,
"client_session_keep_alive": False,
"role": None,
},
}
}
}
mock_get_job_run_artifact.side_effect = get_dbt_artifact
mock_operator.task_id = TASK_ID
mock_operator.run_id = 188471607
mock_task_instance = MagicMock()
mock_task_instance.task_id = TASK_ID
mock_task_instance.dag_id = DAG_ID
mock_client = MagicMock()
mock_client.emit.side_effect = emit_event
mock_get_openlineage_listener.return_value.adapter.get_or_create_openlineage_client.return_value = (
mock_client
)
mock_build_task_instance_run_id.return_value = TASK_UUID
> generate_openlineage_events_from_dbt_cloud_run(mock_operator, task_instance=mock_task_instance)
tests/providers/dbt/cloud/utils/test_openlineage.py:157:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/site-packages/airflow/providers/dbt/cloud/utils/openlineage.py:161: in generate_openlineage_events_from_dbt_cloud_run
client.emit(event=event)
/usr/local/lib/python3.8/unittest/mock.py:1081: in __call__
return self._mock_call(*args, **kwargs)
/usr/local/lib/python3.8/unittest/mock.py:1085: in _mock_call
return self._execute_mock_call(*args, **kwargs)
/usr/local/lib/python3.8/unittest/mock.py:1146: in _execute_mock_call
result = effect(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
event = RunEvent(eventTime='2022-11-10T20:23:55.067332Z', producer='https://github.com/apache/airflow/tree/providers-openlinea...setFacet.json#/$defs/DocumentationDatasetFacet', _deleted=None, description='A starter dbt model')}, outputFacets={})])
def emit_event(event):
> assert event.run.facets["parent"].run["runId"] == TASK_UUID
E TypeError: 'Run' object is not subscriptable
```
Like it's test expectation that is wrong
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage 1.15.0 breaks dbt openlineage integration when running on top of Airlfow 2.9.1 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #39854: OpenLineage 1.15.0 breaks dbt openlineage integration when running on top of Airlfow 2.9.1
URL: https://github.com/apache/airflow/issues/39854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage 1.15.0 breaks dbt openlineage integration when running on top of Airlfow 2.9.1 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39854:
URL: https://github.com/apache/airflow/issues/39854#issuecomment-2132299549
It's actuelly reproducible in main even without 2.9.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage 1.15.0 breaks dbt openlineage integration when running on top of Airlfow 2.9.1 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39854:
URL: https://github.com/apache/airflow/issues/39854#issuecomment-2132299271
That's the change vs. regular PRs where it works fine:
```
< openlineage-integration-common==1.14.0
< openlineage-python==1.14.0
< openlineage_sql==1.14.0
---
> openlineage-integration-common==1.15.0
> openlineage-python==1.15.0
> openlineage_sql==1.15.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org