[GitHub] [incubator-pinot] haibow opened a new pull request #4954: Support schema evolution for consuming segments

[GitBox <gi...@apache.org>]

haibow opened a new pull request #4954: Support schema evolution for consuming segments
URL: https://github.com/apache/incubator-pinot/pull/4954
 
 
   Implement the design in https://github.com/apache/incubator-pinot/issues/4225#issuecomment-548173120
   
   Support querying newly added columns in the consuming segment (OFF by default to keep default behavior the same). Currently after updating the schema for a realtime table and calling reload on the table, consuming segments would not be reloaded with the new schema, which would cause consuming segments to be dropped at query time due to schema inconsistency. 
   
   In this diff, we use the virtual column provider to provide default null values for the newly added columns in the consuming segment (until it is committed), to address the schema inconsistency issue. There is still a delay in indexing and querying the actual values in the newly added columns (up to a few hours, depending on the segment commit thresholds). We will revisit this later.
   
   This functionality is off by default for now to keep the default behavior the same.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org