[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6843: Allow BYTES/Struct and complex type fields to exceed default max length

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on pull request #6843:
URL: https://github.com/apache/incubator-pinot/pull/6843#issuecomment-829743240


   We can do that. My concern around that is we usually store serialized data structure (such as HyperLogLog, TDigest etc.) as BYTES, and truncating the bytes will cause it unable to deserialize.
   I believe the reason why we want to block long strings here is that we use the longest string length for each entry in the string dictionary. Instead, we should automatically switch to var-length string dictionary when we detect strings of very different length.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org