[GitHub] [incubator-pinot] pabrahamusa commented on issue #6317: Allow No Raw Data Index with S3 data store option for TEXT Index

[GitBox <gi...@apache.org>]

pabrahamusa commented on issue #6317:
URL: https://github.com/apache/incubator-pinot/issues/6317#issuecomment-743384566


   @fx19880617   Tiered Storage plus Lazy loading will surely help. However if a raw index is considerably small we can preferably store it in SSDs and separate the data out to object storage. The only caveat here is how to map the index to data locations in object store.  There should be some way to mount the object store locally to pull the corresponding data. Either by defining partitions based on the index which helps to locate the object directly. Also there is a possibility of introducing zstd to further reduce the size and increase the speed of transfer.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org