You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "tirkarthi (via GitHub)" <gi...@apache.org> on 2025/02/07 19:18:51 UTC
[I] Limit and filters not applied correctly for dag runs API. [airflow]
tirkarthi opened a new issue, #46572:
URL: https://github.com/apache/airflow/issues/46572
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
It seems the `limit` filter is not honored in the dagruns api as below. Though limit 14 is applied all dagruns are returned. This is slightly tricky to reproduce since the limit is passed correctly to API and doesn't work through UI but curl works. Ref https://github.com/apache/airflow/pull/46504#discussion_r1945247863
curl http://localhost:8000/public/dags/tutorial_taskflow_api/dagRuns\?limit\=14
### What you think should happen instead?
_No response_
### How to reproduce
1. Visit the dags page with more than 14 runs and the filter is not honored.
2. curl http://localhost:8000/public/dags/tutorial_taskflow_api/dagRuns\?limit\=14
### Operating System
Ubuntu 20.04.3 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Limit and filters not applied correctly for dag runs API. [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46572:
URL: https://github.com/apache/airflow/issues/46572#issuecomment-2645999623
From your initial PR conversation, I understood you are referencing the part where you extract `total_entries` from the response, and it shows all `total_entries` despite the limit set to 14.
If so I believe that this number is not affected by pagination limits and offsets, the `total_entries` would be based on count from `filters=` fields.
```python
dag_run_select, total_entries = paginated_select(
statement=query,
filters=[logical_date, start_date_range, end_date_range, update_at_range, state],
order_by=order_by,
offset=offset,
limit=limit,
session=session,
)
```
And with curl behavior is also the same:
<details>
<summary>curl run for 2 entries</summary>
```bash
curl -X 'GET' \
'http://localhost:29091/public/dags/example_bash_decorator/dagRuns?limit=2&order_by=-start_date' \
-H 'accept: application/json' | jq
{
"dag_runs": [
{
"dag_run_id": "manual__2025-02-08T23:20:51.665298+00:00",
"dag_id": "example_bash_decorator",
"logical_date": "2025-02-08T23:20:51.673703Z",
"queued_at": "2025-02-08T23:20:51.688648Z",
"start_date": "2025-02-08T23:38:06.692506Z",
"end_date": "2025-02-08T23:38:09.660110Z",
"data_interval_start": "2025-02-08T23:20:51.673703Z",
"data_interval_end": "2025-02-08T23:20:51.673703Z",
"last_scheduling_decision": "2025-02-08T23:38:09.656040Z",
"run_type": "manual",
"state": "failed",
"external_trigger": true,
"triggered_by": "rest_api",
"conf": {},
"note": null
},
{
"dag_run_id": "manual__2025-02-08T23:04:32.759689+00:00",
"dag_id": "example_bash_decorator",
"logical_date": "2025-02-08T23:04:32.759689Z",
"queued_at": "2025-02-08T23:04:32.784971Z",
"start_date": "2025-02-08T23:04:33.602888Z",
"end_date": "2025-02-08T23:04:36.819802Z",
"data_interval_start": "2025-02-08T23:04:32.759689Z",
"data_interval_end": "2025-02-08T23:04:32.759689Z",
"last_scheduling_decision": "2025-02-08T23:04:36.817639Z",
"run_type": "manual",
"state": "failed",
"external_trigger": true,
"triggered_by": "ui",
"conf": {},
"note": null
}
],
"total_entries": 127
}
```
</details>
Or the problem is that API returns the wrong number of entries in `dag_runs` field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Limit and filters not applied correctly for dag runs API. [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #46572:
URL: https://github.com/apache/airflow/issues/46572#issuecomment-2647644572
I have identified the problem.
I should be able to open a PR to fix that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AIP-84 Limit and filters not applied correctly for dag runs API. [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun closed issue #46572: AIP-84 Limit and filters not applied correctly for dag runs API.
URL: https://github.com/apache/airflow/issues/46572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AIP-84 Limit and filters not applied correctly for dag runs API. [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #46572:
URL: https://github.com/apache/airflow/issues/46572#issuecomment-2647695004
Thanks @pierrejeambrun , from my understanding the QueryLimit and QueryOffset are module level variable initialised only once and using it on type annotations the same LimitFilter object gets used for requests. This is reproducible only in UI with parallel requests where the limit values of different APIs overwrite each other and the final value is taken into account. In curl since it's sequential the limit filter object is updated properly. This might be applicable to other filters as well constructed in similar manner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AIP-84 Limit and filters not applied correctly for dag runs API. [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #46572:
URL: https://github.com/apache/airflow/issues/46572#issuecomment-2647859752
PR: https://github.com/apache/airflow/pull/46619
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AIP-84 Limit and filters not applied correctly for dag runs API. [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46572:
URL: https://github.com/apache/airflow/issues/46572#issuecomment-2647735327
Sorry for my previous question, but the exact problem was initially unclear to me.
Yesterday I was able to reproduce the behavior you described now with async requests to API with a single HTTP session, but didn't have time to explain what I found.
Great to see that the root cause was identified already by Pierre!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org