[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #7153: New data keep saving in old format after changing schema

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on issue #7153:
URL: https://github.com/apache/incubator-pinot/issues/7153#issuecomment-878412701


   There are 2 ways of handling the schema change with new columns within consuming segment:
   1. (Supported) Keep the already consumed records and add new columns as default values. The next consuming segment will pick up the new schema and consume the actual data. With this approach, the consumed records are kept, but we have to wait for the next consuming segment to ingest the actual data for the new columns (we might be able to build an api to force commit the current consuming segments and create new consuming segments)
   2. (Not supported yet) Drop the already consumed records and re-consume from the start offset with the new columns with actual data. This approach can load data for new columns immediately, but the already consumed records are dropped and re-consumed which can cause temporary data inconsistency.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org