[GitHub] [incubator-pinot] chenboat opened a new pull request #4914: [POC] By-passing deep-store requirement for Realtime segment completion

[GitBox <gi...@apache.org>]

chenboat opened a new pull request #4914: [POC] By-passing deep-store requirement for Realtime segment completion
URL: https://github.com/apache/incubator-pinot/pull/4914
 
 
   Design doc and context [[doc](https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion)].
   
   This PR serves as a complete POC for the design. I may break it into smaller PR in future. It contains a few key parts:
   (1) a new Pinot server rest api to allow download of a zip'ed segment.
   (2) Modifief LLC segment protocol which allows the committing server to skip synchronous segment upload to the deep storage (via controller or otherwise). Instead only meta data is uploaded. The segment upload becomes optional. 
   (3) Servers may now download segments from peer servers instead.
   (4) Two new integration tests to demo how things work end to end with and without deep storage.
   
   @jamesyfshao 
    

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org