[GitHub] [incubator-pinot] chenboat commented on pull request #6567: Enable segment upload to a realtime table

[GitBox <gi...@apache.org>]

chenboat commented on pull request #6567:
URL: https://github.com/apache/incubator-pinot/pull/6567#issuecomment-776953396


   > How do we manage the stream offset if we upload a segment to the realtime table? Also, how about the segment name format?
   > We need to have an integration test to ensure the `PinotLLCRealtimeSegmentManager` can work properly on uploaded segments.
   
   Please refer to the comments I added with respect to Subbu's questions. For upsert tables, there should be validation that the offset of uploaded segments will not affect the consuming segment's offset. This is inline with the goals of segment upload — which are mainly about bootstrapping historical data. 
   
   The segment name format I propose conforms to the LLC name format with the following parts. tableName(string)_partitionId_(int)_sequenceNumber(int)_creationTime(string). The main difference is that the creationTime string now does not need to an actual timestamp but rather any unique string (e.g., timestamp + an uuid).
   
   Yes. I will add an integration test.
    
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org