You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/06/12 18:14:38 UTC
[GitHub] [incubator-pinot] pradeepgv42 opened a new issue #5554: Using deep storage as a tiered storage
pradeepgv42 opened a new issue #5554:
URL: https://github.com/apache/incubator-pinot/issues/5554
Currently IIUC all the segments need to be on disk to be able to query them.
Usually, queries to older data tends to be lower or clients would be okay with
in-curing latency for queries on older data.
Using this assumption, we could possibly keep the segments older than X days
on deep storage and not load them onto the servers.
Couple of ways to deal with this based on discussions on slack:
1) Using presto on top of pinto and make presto-pinot be able to query
segments on S3 directly.
2) When the query arrives lazy load the segment and return the results.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on issue #5554: Using deep storage as a tiered storage
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #5554:
URL: https://github.com/apache/incubator-pinot/issues/5554#issuecomment-650048355
This would be a good addition.
We can potentially build a storage hierarchy as DRAM -> SSD/HDD -> Remote store (Deep Store).
- Segments are adaptively (based on the query pattern) brought onto the compute nodes (servers) and stored on local storage (SSD or HDD).
- Furthermore, heavily queried (and based on size) segments can be cached completely in DRAM on the servers. This would be helpful for systems that don't have SSD and thus paging overhead for memory mapped segments is non-trivial as compared to SSD.
Azure Kusto implements this hierarchy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org