You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "collinmcnulty (via GitHub)" <gi...@apache.org> on 2025/04/28 15:19:36 UTC
[I] Idempotency for Event Scheduled DAGs [airflow]
collinmcnulty opened a new issue, #49900:
URL: https://github.com/apache/airflow/issues/49900
### Description
There should be an option on the AssetWatcher to respect idempotency in the payload of a TriggerEvent could include when yielded, so that a new DAG is only created when a new key shows up. This is already the case for task sensors for HA purposes, so the DAG creation logic surely could respect that too.
This way you could make S3KeyTrigger and similar triggers work by having them put the name of the key that was detected (or modified time if the key is fully defined by the trigger) and put that in the TriggerEvent payload. Then only one DAG would run per time the condition became true, rather than scheduling infinitely.
### Use case/motivation
I was looking to turn a DAG into one with event-driven scheduling and the [notice on infinite scheduling](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/event-scheduling.html#avoid-infinite-scheduling) gave me real pause. It seems like the trigger itself needs to consume the message, whereas my expectation is that most data engineers would find it more natural that the trigger would detect the condition and the running of the DAG would cause the condition to stop being true. E.g. trigger detects a file in a location and the DAG does something with the file then deletes it. But instead what happens is that the [trigger has to delete the file](https://github.com/apache/airflow/blob/main/providers/standard/src/airflow/providers/standard/triggers/file.py#L127).
### Related issues
#49857 would also be fixed by this. Let files pile up in a location with the DAG paused, turn the DAG back on, and suddenly you get an event per file.
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Idempotency for Event Scheduled DAGs [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #49900:
URL: https://github.com/apache/airflow/issues/49900#issuecomment-2835888029
> The trigger retrieves a list of all objects in the object store with a “last updated” timestamp >= timestamp of the “last Asset Event”.
Just a note. S3 has a `start_after_key` feature so it is optimal, for incremental processes from s3, if the keys are lexicographically sorted in order of arrival -- this way we do not need to list the whole directory and filter by timestamp in python (which can be slow and costly). Unless s3 changed, it does not have an analogous timestamp filter capability.
Anyway, so tthen each run, you pass "last key" to `start_after_key` in s3 api and you get only the new fiiles.
Other blob storages may or may not have similar feature.
So we should not necessarily get tunnel vision on timestamps -- because they are not always optimal.
Also I personally like to avoid using the word idempotent in Airflow discussions because it's not always clear exactly what we mean by that. It can mean different things in different contexts and I think clearer, more precise language is usually available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Idempotency for Event Scheduled DAGs [airflow]
Posted by "collinmcnulty (via GitHub)" <gi...@apache.org>.
collinmcnulty commented on issue #49900:
URL: https://github.com/apache/airflow/issues/49900#issuecomment-2835853006
From private convo with @dstandish : Maybe you could take advantage of the fact that run_id is unique and generate the run_id based on the payload of the TriggerEvent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Idempotency for Event Scheduled DAGs [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #49900:
URL: https://github.com/apache/airflow/issues/49900#issuecomment-2835642538
This is definitely that I've been thinking about a bit. We're going to have to some sort of a "time-based" approach to this. I've drafted up some thoughts here; this is WIP.
### Proposed Solution
- An `ObjectStore` (S3, WASB, etc.) AssetWatcher is written, and leverages some Trigger. Really, this Trigger will be doing most (all) of the work.
- This Trigger pulls from the `asset` endpoint to determine the `last_asset_event_timestamp` for that asset (see "Changes Needed" below, I am close to closing this issue out).
- The trigger retrieves a list of all objects in the object store with a “last updated” timestamp `>= timestamp` of the “last Asset Event”.
- For each of these resulting objects, an Asset Event is written to the `asset_event` endpoint (using timestamp that the object itself landed/was updated in the object store), and yielded to trigger a DAG Run. *Note that the implementation of #47164 will include a write to the `/assets` endpoint to update the `last_asset_event_id` and `last_asset_event_timestamp`.
### Changes Needed
Outside of the general development of this `ObjectStore` AssetWatcher, a few other changes are needed in order to implement this pattern.
- The addition of a `last_asset_event` field to the `asset` endpoint. See #47164 . I chatted with Pierre; he implied that this would be the desired way to go about this, so I’m going to go ahead and add to the issue.
## Assumptions/Constraints
I’m making a number of assumptions/constraints in order to come to the proposed solution above. These include the following:
1. A single object store is considered an Asset.
2. Each time that a object lands in an object store or is updated, it should be considered an Asset Event. Each file object is NOT an Asset itself in this model (see below).
3. There is a fundamental difference between watching an object store, and watching an object. These should be two different `AssetWatcher`'s.
4. When an `AssetWatcher` runs, it should not alter the state of the object store.
5. It is untenable to “scan” the entire object store each time the `AssetWatcher` runs. This would be too computationally expensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org