[GitHub] [incubator-pinot] chenboat commented on issue #4741: Make the download url of a realtime LLC segment as a list of urls.

[GitBox <gi...@apache.org>]

chenboat commented on issue #4741: Make the download url of a realtime LLC segment as a list of urls.
URL: https://github.com/apache/incubator-pinot/pull/4741#issuecomment-546154132
 
 
   @Jackie-Jiang @mcvsubbu First thank you for the comments and proposals.  I have updated the design doc with a new[ section](https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion#By-passingdeep-storerequirementforRealtimesegmentcompletion-SegmentDownload
   ) to capture the key issue above discussion so far. Please comment there if you have further issues. 
   
   As for Jackie's points, 
    - the update to Zk metadata need NOT be done by servers: server can invoke controller API to do the update. 
   
   - The more important issue is how to discover such server list, I do agree that using External View is better (i remember we briefly discussed about this idea as well). We also need to double check all current download processes using the download_url can fallback to use EV to download from peer servers.
   
   - For segment download format, currently the commit server already Gzips its segment and then upload to the controller as a single file. The new download REST API on the Pinot server uses the same process and also sends such GZip'd file to the server which needs the segment. Therefore Gzip compression is mainly for REST download easiness. I am not sure if there is an easy not to Gzip a segment here. I am not sure if we want to do too much optimization here -- given segment download in LLC is in fact a rare event and the change is further reduced by available deep storage copies.       
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org