[GitHub] [incubator-pinot] siddharthteotia commented on pull request #6710: DataTable V3 implementation and measure data table serialization cost on server

[GitBox <gi...@apache.org>]

siddharthteotia commented on pull request #6710:
URL: https://github.com/apache/incubator-pinot/pull/6710#issuecomment-809610152


   > @mcvsubbu @siddharthteotia and me meet offline, we wanner keep this PR focus on bumping up to v3 and move metadata to the end of data table, also use enmu ordinal as key when serialize. And make it configurable to send V2/V3 data at server side (instance config).
   > 
   > @Jackie-Jiang I terms of addressing the TODO in DataTableBuilder(fix float data length, one String->Int map for the whole table instead of for each column), we will address it separately(bumping up to V4).
   
   Yes, the enum based original approach is much simpler and the enum getName() gives the camel case string key and the enum ordinal gives the id. The approach of using explicit id and name is not needed since we want the enum structure to always grow - users should not be allowed to remove enums or change name of enums. Should only be adding new enums to the end. 
   
   For the existing TODOs, the main reason for not addressing them in this PR is to keep this as simple as possible. It is easier  to review and once this change goes in production and if there are any issues, we don't have to debug multiple independent changes. We can always address those TODOs in a follow-up and if they demand version change, we can bump the version again.
   
   Regarding introducing a instance config for version, we agreed that CURRENT_VERSION will be 3 and server will always send the data table with this version. @mcvsubbu  suggested that it can be helpful to force server to send the old version v2 in case there are issues with v3 forcing a rollback but we want to use the latest code. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org