You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "theamit45 (via GitHub)" <gi...@apache.org> on 2025/01/24 20:23:45 UTC
[I] the List APIs return all available records in a single response. This approach is inefficient and can lead to performance issues, particularly when dealing with large datasets. The lack of pagination makes it difficult. [iceberg]
theamit45 opened a new issue, #12099:
URL: https://github.com/apache/iceberg/issues/12099
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] The List APIs currently lack pagination functionality, resulting in inefficient data retrieval for large datasets. When retrieving a substantial number of records, the absence of pagination leads to performance bottlenecks such as slow response times, increased memory usage, and potential server crashes. To improve the usability, scalability, and performance of the APIs, it is essential to implement pagination. [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer closed issue #12099: The List APIs currently lack pagination functionality, resulting in inefficient data retrieval for large datasets. When retrieving a substantial number of records, the absence of pagination leads to performance bottlenecks such as slow response times, increased memory usage, and potential server crashes. To improve the usability, scalability, and performance of the APIs, it is essential to implement pagination.
URL: https://github.com/apache/iceberg/issues/12099
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] The List APIs currently lack pagination functionality, resulting in inefficient data retrieval for large datasets. When retrieving a substantial number of records, the absence of pagination leads to performance bottlenecks such as slow response times, increased memory usage, and potential server crashes. To improve the usability, scalability, and performance of the APIs, it is essential to implement pagination. [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #12099:
URL: https://github.com/apache/iceberg/issues/12099#issuecomment-2613362984
Closing this as another GenAI issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] The List APIs currently lack pagination functionality, resulting in inefficient data retrieval for large datasets. When retrieving a substantial number of records, the absence of pagination leads to performance bottlenecks such as slow response times, increased memory usage, and potential server crashes. To improve the usability, scalability, and performance of the APIs, it is essential to implement pagination. [iceberg]
Posted by "theamit45 (via GitHub)" <gi...@apache.org>.
theamit45 commented on issue #12099:
URL: https://github.com/apache/iceberg/issues/12099#issuecomment-2613324335
### Context
- **Current Behavior:** List APIs return all records in a single response without pagination.
- **Observed Issues:** Large datasets cause performance issues, including slower response times, increased memory consumption, and higher server load.
- **User Impact:** Developers consuming these APIs face difficulties in efficiently handling and processing large responses, requiring additional client-side logic to mitigate issues.
### Expected Behavior
- Implement pagination in List APIs to limit the number of records returned per response.
- Provide query parameters like `page`, `limit`, or `offset` for clients to specify pagination preferences.
- Include metadata in the response, such as `total records`, `current page`, and `total pages`, to improve usability.
### Proposed Solution
1. Add support for pagination to all List APIs using a standard approach (e.g., offset-based or cursor-based pagination).
2. Update the response format to include pagination metadata (e.g., `page`, `limit`, `total_records`, `total_pages`).
3. Ensure backward compatibility by maintaining the current behavior for clients that do not provide pagination parameters.
4. Update documentation to reflect the new pagination functionality and include examples of paginated API usage.
5. Add test cases to validate the correctness of paginated responses, covering edge cases like empty datasets, invalid page numbers, and large datasets.
### Benefits
- **Improved Performance:** Pagination reduces the response size and optimizes server resource usage.
- **Scalability:** APIs become more scalable and maintain performance with increasing dataset sizes.
- **Better Developer Experience:** Developers can handle data more effectively, with less need for client-side workarounds.
### Additional Considerations
- Benchmark the performance of different pagination strategies to choose the most efficient implementation.
- Ensure that paginated APIs work seamlessly with other query parameters, such as filters and sorting.
---
Was this feedback helpful?
**Reviewer ID:** 0c6cda
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org