Apache Pinot Daily Email Digest (2020-10-25)

[Pinot Slack Email Digest <sn...@apache.org>]

### _#general_

  
 **@noahprince8:** @noahprince8 has joined the channel  
 **@noahprince8:** How does Pinot scale with offline tables? I get the
impression that every offline segment is loaded into an active offline server,
which implies _all_ of your offline data is loaded in some server. This seems
very expensive, especially for something like 2 year old data. Does pinot
lazily load old segments based on query demand? And how do indexes scale into
offline tables?  
**@mayanks:** We mmap the indexes, so they get paged in as needed. Depending
on your sla requirements, you can use SSD or regular HDD on server nodes  
**@noahprince8:** I'm talking more for something like using s3 as offline
access  
**@noahprince8:** I'm looking for something that can hit low latency SLAs but
retire old data to s3 daily or weekly. This will be large volumes of data
(200k+ messages/sec), so we can't really be keeping all of it in normal
storage.  
**@noahprince8:** Clickhouse + Parquet files in s3 + Presto is a workable
solution, but doesn't really give you any indexing in offline mode. Pinot
looked interesting in that it might bridge that gap between historical and
real-time querying  
**@g.kishore:** You can use ebs mounted volume  
**@noahprince8:** Right, but price-wise aren't EBS volumes much more expensive
than S3 storage?  
**@g.kishore:** Yes,  
**@g.kishore:** We don’t have native s3 support as of now -  
**@yupeng:** You can think of Pinot as an indexing engine, so you can index
the fields that you will query. If you want to explore on demand caching,
there is no such thing in Pinot yet. However, you can explore other file
system caching service like alluxio, and mount s3 as underneath storage for
Pinot.  
**@yupeng:** Btw I have not tried this Alluxio set up with Pinot. Though in
theory it works, you might have to investigate  

### _#random_

  
 **@noahprince8:** @noahprince8 has joined the channel  

###  _#troubleshooting_

  
 **@sk23.osc:** @sk23.osc has joined the channel  

###  _#release060_

  
 **@kharekartik:** @kharekartik has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org