You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "phi-friday (via GitHub)" <gi...@apache.org> on 2024/08/19 03:11:39 UTC
[I] Gets a `DatasetEvent` that was created later than `logical_date`. [airflow]
phi-friday opened a new issue, #41571:
URL: https://github.com/apache/airflow/issues/41571
### Apache Airflow version
2.10.0rc1
### If "Other Airflow 2 version" selected, which one?
2.10.0(not rc)
### What happened?
I was expecting to be able to access the `DatasetEvent` of the currently triggered `DatasetEvent`, but it accesses the last created `Extra`.
### What you think should happen instead?
Only `DatasetEvent` created before `logical_date` should be looked up.
### How to reproduce
```python
from __future__ import annotations
from collections.abc import Generator
from airflow.datasets import Dataset, DatasetAlias
from airflow.datasets.metadata import Metadata
from airflow.decorators import dag, task
from airflow.operators.bash import BashOperator
from airflow.operators.trigger_dagrun import TriggerDagRunOperator
from pendulum import datetime
dataset_alias = DatasetAlias("my_dataset_alias")
dataset = Dataset("my_dataset")
@dag(start_date=datetime(2024, 1, 1), schedule=None, catchup=False)
def dataset_producer() -> None:
@task.python(outlets=[dataset_alias])
def with_extra() -> Generator[Metadata, None, None]:
import json
from airflow.operators.python import get_current_context
context = get_current_context()
conf = dict(context["dag_run"].conf)
conf = json.dumps(conf)
yield Metadata(
target="my_dataset",
extra={"key": "value", "conf": conf},
alias="my_dataset_alias",
)
with_extra()
@dag(start_date=datetime(2024, 1, 1), schedule=None, catchup=False)
def dataset_producer_twice() -> None:
first = TriggerDagRunOperator(
task_id="first",
trigger_dag_id="dataset_producer",
deferrable=False,
wait_for_completion=False,
conf={"task_id": "first"},
)
second = TriggerDagRunOperator(
task_id="second",
trigger_dag_id="dataset_producer",
deferrable=False,
wait_for_completion=False,
conf={"task_id": "second"},
)
wait = BashOperator(task_id="wait", bash_command="sleep 5")
_ = [first, wait] >> second
@dag(start_date=datetime(2024, 1, 1), schedule=dataset_alias, catchup=False)
def dataset_consumer() -> None:
wait = BashOperator(task_id="wait", bash_command="sleep 10")
@task.python(inlets=[dataset_alias])
def consume_dataset() -> None:
from pprint import pprint
from airflow.datasets import DatasetAlias
from airflow.operators.python import get_current_context
context = get_current_context()
inlet_events = context["inlet_events"]
target_events = inlet_events[DatasetAlias("my_dataset_alias")]
event = target_events[-1]
pprint(event.extra)
show = consume_dataset()
_ = wait >> show
dataset_producer()
dataset_consumer()
dataset_producer_twice()
```
You can see the same logs in both `DagRun`s.
```[2024-08-19, 12:06:20 KST] {logging_mixin.py:190} INFO - {'conf': '{"task_id": "second"}', 'key': 'value'}```
### Operating System
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.7.3
apache-airflow-providers-common-compat==1.1.0
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.15.0
apache-airflow-providers-docker==3.12.3
apache-airflow-providers-fab==1.2.2
apache-airflow-providers-ftp==3.10.1
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-jdbc==4.4.0
apache-airflow-providers-odbc==4.6.3
apache-airflow-providers-postgres==5.11.3
apache-airflow-providers-redis==3.7.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-sqlite==3.8.2
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Gets a `DatasetEvent` that was created later than `logical_date`. [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday closed issue #41571: Gets a `DatasetEvent` that was created later than `logical_date`.
URL: https://github.com/apache/airflow/issues/41571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Gets a `DatasetEvent` that was created later than `logical_date`. [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #41571:
URL: https://github.com/apache/airflow/issues/41571#issuecomment-2295595760
This is solved by using `triggering_dataset_events`. 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org