Apache Pinot Daily Email Digest (2020-10-22)

[Pinot Slack Email Digest <sn...@apache.org>]

### _#general_

  
 **@darren:** @darren has joined the channel  
 **@darren:** Hiiiiii all . :+1: :heart: :slightly_smiling_face:  
**@fx19880617:** hello  
 **@gilli.aswin:** @gilli.aswin has joined the channel  
 **@hisham.itani:** for those interested, there's a Pinot subreddit  
**@ho0001vu:** @ho0001vu has joined the channel  
 **@npawar:** Hey folks, @chinmay.cerebro and @baliga from the community will
be speaking at Flink Forward, which is happening today! They will talk about
how Apache Pinot and Apache Flink power realtime analytics at Uber. Their
session is at 12.50pm PDT. Please tune in if you’re around!  
**@brianjstafford:** @brianjstafford has joined the channel  

###  _#random_

  
 **@darren:** @darren has joined the channel  
 **@gilli.aswin:** @gilli.aswin has joined the channel  
 **@ho0001vu:** @ho0001vu has joined the channel  
 **@brianjstafford:** @brianjstafford has joined the channel  

###  _#announcements_

  
 **@muralidharanece:** @muralidharanece has joined the channel  

###  _#metadata-push-api_

  
 **@steotia:** Hi, unrelated to the metadata push API and the recent
discussion on table lock: Are there any thoughts on the following for current
non-deepstore based segment push to controller: • getting rid of untarring
business by doing a multi-part push comprising of segment data and metadata
separately • controller has to untar everything, fish out metadata and then do
CRC check etc. If the metadata is provided separately, it doesn't have to do
anything. • Our localTemp space for controllers is on HDD and we are seeing
30MB/sec there  
 **@steotia:** @mayanks @fx19880617  
 **@mayanks:** We can add optional metadata along with payload  
**@mayanks:** And use SSD  
 **@steotia:** exactly  
 **@mayanks:** Will that solve our problem  
**@steotia:** SSD for sure if we are not able to get rid of untarring business  
 **@fx19880617:** I somehow feel the bottleneck is the controller download, if
we can bypass this path, then controller overhead will be reduced a lot  
 **@mayanks:** Yes controller bypassing will help a lot  
 **@mayanks:** Unfortunately at lnkd we don’t have a reliable deepstore  
 **@fx19880617:** we can add an option to convert metadata.properties and
creation.meta file to json and let controller to handle that as well  
 **@fx19880617:** basically there is no need for controller to download the
data part  
 **@fx19880617:** even current just metadata tar file helps a lot on the
controller side  
 **@mayanks:** But we don’t have a deepstore that is reliable and can be used
in prod  
**@mayanks:** So we have no way to bypass controller  
 **@fx19880617:** ic  
 **@fx19880617:** not even hdfs?  
 **@fx19880617:** pinot segments are anyway generated there and stored right?  
 **@mayanks:** Availability is not good enough for production  
**@fx19880617:** ic  
 **@fx19880617:** even offline push is not good ?  
 **@mayanks:** Offline push is good  
 **@fx19880617:** for realtime side  
 **@fx19880617:** Uber has implemented the p2p download  
 **@fx19880617:** maybe you wanna try out  
 **@mayanks:** So we are trying out ideas for deepstore  
**@fx19880617:** so servers can download from each other  
 **@fx19880617:** got it  
 **@mayanks:** But in case none of those work we want to still optimize with
controller in the path  
 **@fx19880617:** maybe 2 hdfs :slightly_smiling_face:  
 **@fx19880617:** hdfs1:// and hdfs2://  
 **@fx19880617:** I remember at Uber they used to use two hdfs to satisfy the
availability requirements :stuck_out_tongue:  
 **@mayanks:** :smiley:  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org