You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pierrejeambrun (via GitHub)" <gi...@apache.org> on 2026/04/17 09:09:30 UTC
[PR] [v3-2-test] Add cursor based pagination for get_task_instances endpoint (#64845) [airflow]
pierrejeambrun opened a new pull request, #65405:
URL: https://github.com/apache/airflow/pull/65405
* Add cursor-based pagination to get_task_instances endpoint
- Add cursor-based (keyset) pagination as an alternative to offset-based pagination on the get_task_instances endpoint. Offset pagination remains the default and is not deprecated globally.
- Response uses a discriminated union: offset responses include total_entries, cursor responses include next_cursor and previous_cursor.
- Refactor SortParam to lazily cache column resolution instead of mutating state in to_orm.
- Move cursor helpers (encode/decode/apply) to dedicated common/db/cursors.py module.
- Cleanly separate cursor vs offset code paths in the endpoint handler.
* Simplify cursor token and support first page without sentinel
- Remove order_by from cursor token (now just a list of values)
- Support empty string cursor for first page (no fake sentinel needed)
- Drop order_by consistency check between cursor and query param
* Small adjustments
* Adjustments
* Narrow endpoint return types and encode cursor value types
Encode type information directly into cursor tokens as {"type": ..., "value": ...} objects, removing the fragile column-based type guessing during deserialization.
Narrow return types for endpoints that only return offset pagination (patch, clear, batch, mapped) so the OpenAPI spec and generated UI client reflect the correct types. Only get_task_instances retains the discriminated union response.
Update UI components to use the narrowed types from the spec.
* Use msgpack for cursor tokens and nested keyset predicate
Switch cursor encoding from typed JSON to msgpack for compactness. Replace flat OR-of-prefix-equalities with nested and/or keyset predicate for better composite index range scans. Always use ascending PK as the final tie-breaker for stable pagination.
* Flatten TaskInstanceCollectionRes ponse to avoid oneOf codegen issues
Replace the discriminated union (offset | cursor response types) with a single flat model using optional fields. OpenAPI oneOf + discriminator is not handled correctly by hey-api/openapi-ts (#1613, #3270): return types degrade to unknown in generated TypeScript code.
* Fix UI
* Fix CI
* Fix cursor pagination boundary detection and error handling
- Fetch limit+1 rows to accurately detect last page, returning next_cursor=null when no more results exist
- Return previous_cursor=null on the first page (when no cursor was provided)
- Use LimitFilter in apply_filters_to_select for the +1 limit instead of a manual .limit() call
- Raise HTTP 400 on invalid UUID in cursor token instead of silently passing the invalid value
- Update endpoint docs and add boundary-condition test
* Fix backward cursor based pagination
(cherry picked from commit e11c6037f3a8d53abb8c03e6010469133621929c)
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [v3-2-test] Add cursor based pagination for get_task_instances endpoint (#64845) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #65405:
URL: https://github.com/apache/airflow/pull/65405#issuecomment-4281393788
Unrelated static check failure, merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [v3-2-test] Add cursor based pagination for get_task_instances endpoint (#64845) [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun merged PR #65405:
URL: https://github.com/apache/airflow/pull/65405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org