[GitHub] clintropolis commented on issue #6549: Don't store all-null dimension columns

[GitBox <gi...@apache.org>]

clintropolis commented on issue #6549: Don't store all-null dimension columns
URL: https://github.com/apache/incubator-druid/issues/6549#issuecomment-435261020
 
 
   @leventov do you have something in mind like adding a new 'version' to the dictionary column part header to indicate a constant null dimension if we notice that the dictionary only has null at serialization time, so that way the header of the part is still written, just not the sub parts in order to be able to still distinguish an all null dimension from a dimension that doesn't exist?
   
   With regards to constant elimination (all-zero columns), #6016, does this at the "column chunk scale", by collecting statistics within the chunk in a class called [`IntFormMetrics`](https://github.com/apache/incubator-druid/pull/6016/files#diff-638f4e499d0412cbf444dcc71a2b34aaR34) which [is passed row values as they are sent to the serializer](https://github.com/apache/incubator-druid/pull/6016/files#diff-13c325751ca6de8ad95767e679d37d7eR193).
   
   These statistics are discarded after the chunk is encoded, but longer term, I was considering saving until all chunks were processed, and making additional optimizations to eliminate constant columns almost entirely by encoding that information in the int column header, allowing the column supplier to return specialized/optimized implementations for these situations. I would imagine all null columns would also be relatively cheap represented in this manner, but would have more overhead in the form of a few function calls, since we could i think we could just push all of this up into a specialized selector that always returns nulls and wouldn't have to call into dictionaries/columns/indexes if we have a specialized dictionary column part for nulls like you are suggesting?
   
   Additionally, I was planning to explore if that same information could be used beyond serialization of the int column to advise on index creation, e.g. whether to use concise or roaring (i need to do some experimentation here to see if any patterns pop up that would be good indicators to decide one way or another), or if we add any additional types of indexes in the future.
   
   I think maybe also would be worthwhile to continue the discussion of #5347 since this relates to segment format perhaps? I plan on weighing in on that issue based on my experiences working on that branch in the very near future.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org