You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/05/10 02:32:25 UTC
[GitHub] [pinot] walterddr opened a new issue, #8668: [Feature] support simple cursor operations
walterddr opened a new issue, #8668:
URL: https://github.com/apache/pinot/issues/8668
Background
===
One of the frequently asked features of Pinot is the ability to retrieve the query result in small batches. For example,
- users may perform resource-intense computations on the result on a row-by-row base; and materializing the entire result set in memory is wasteful.
- support for some other systems like sql-alchemy (in https://github.com/python-pinot-dbapi/pinot-dbapi/) requires an engine to implement fetch-one and fetch-many
Challenge
===
currently, Pinot executes these subset-of-result executions via the 2 argument limit clause; for example
```
SELECT * FROM myTbl limit 200,100
```
returns 100 rows of the SQL result, starting from the 200th row.
This poses a problem
1. it doesn't guarantee that the results are stable as the underlying data could've changed / or early termination of the limit kicks in and we have no idea which server returns the result set first.
2. it does the entire query execution again for each subset, e.g. `limit 0, 100` then `limit 100, 100` doesn't retain the previous execution result.
Proposal
===
Support a simple cursor operation by enabling data caching in Pinot broker before serving back to the users.
1. Return results with the cursor-support turned on should associate the result with the requestID being executed.
2. Subsequence cursor operation can reuse the same requestID to seek for the next subset.
3. Pinot broker caches the cursor result set until it times out.
Alternative Solution(s)?
===
Please comment below for any additional alternative solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on issue #8668: [Feature] support simple cursor operations
Posted by GitBox <gi...@apache.org>.
61yao commented on issue #8668:
URL: https://github.com/apache/pinot/issues/8668#issuecomment-1341916261
Caching the result in broker means caching in memory or spilling to disk?
I feel caching in memory is dangerous for broker. Have we considered storing the result in an external storage other than broker?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on issue #8668: [Feature] support simple cursor operations
Posted by GitBox <gi...@apache.org>.
atris commented on issue #8668:
URL: https://github.com/apache/pinot/issues/8668#issuecomment-1121926797
As discussed, I will taking a shot at this.
@Jackie-Jiang Any thoughts/inputs to share, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #8668: [Feature] support simple cursor operations
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #8668:
URL: https://github.com/apache/pinot/issues/8668#issuecomment-1188509727
I gave a comment regarding this on a related issue - https://github.com/apache/pinot/issues/5246
PTAL. cc @jackjlli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jackjlli commented on issue #8668: [Feature] support simple cursor operations
Posted by GitBox <gi...@apache.org>.
jackjlli commented on issue #8668:
URL: https://github.com/apache/pinot/issues/8668#issuecomment-1341790740
Hi all,
The proposed design doc on pagination support has been linked to this issue: https://github.com/apache/pinot/issues/5246.
Please review it and feel free to put any comments. Thanks!
Best,
Jack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org