You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/05/21 05:53:59 UTC

[GitHub] [pinot] Jackie-Jiang opened a new issue, #8755: Support raw index for multi-valued columns

Jackie-Jiang opened a new issue, #8755:
URL: https://github.com/apache/pinot/issues/8755

   Related issues: #6473, #8750 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] suddendust commented on issue #8755: Support raw index for multi-valued columns

Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #8755:
URL: https://github.com/apache/pinot/issues/8755#issuecomment-1328253645

   @Jackie-Jiang IIRC, we have a very high cardinality MV column on which we want to turn off dict encoding. However, do so does not let us define inverted index on these columns. So we want to be able to have the inverted index WHILE having the option to turn off the dictionary for this column. The dictionary ends up occupying around 50% of the segment size.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] suddendust commented on issue #8755: Support raw index for multi-valued columns

Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #8755:
URL: https://github.com/apache/pinot/issues/8755#issuecomment-1328253905

   @Jackie-Jiang Further description [here](https://github.com/apache/pinot/issues/8750).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on issue #8755: Support raw index for multi-valued columns

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8755:
URL: https://github.com/apache/pinot/issues/8755#issuecomment-1322872150

   Don't remember the intention for this issue, but the raw index for MV columns is already supported with #7595


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on issue #8755: Support raw index for multi-valued columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #8755:
URL: https://github.com/apache/pinot/issues/8755#issuecomment-1134952819

   hey @Jackie-Jiang - there is another related issue here https://github.com/apache/pinot/issues/7870
   
   I had discussed the point (4) in that issue with @somandal  and we are planning to do it in one of the two ways:
   
   - Support for general purpose compression on dictionary encoded columns. So, for a dictionary encoded column it is not absolutely necessary to dictionary encode the forward index as well. The raw data in forward index can be compressed using LZ4 / SNAPPY etc just like we would have done for a noDictionary column
   - The other way we are thinking is to also add support for no forward index.
   
   I think both approaches can be implemented and will benefit the case where forward index can be huge (like the MV column example in my issue) and the column is only used in filtering so ok to have the forward index compressed or not have it at all. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] suddendust commented on issue #8755: Support raw index for multi-valued columns

Posted by GitBox <gi...@apache.org>.
suddendust commented on issue #8755:
URL: https://github.com/apache/pinot/issues/8755#issuecomment-1318350823

   @siddharthteotia Any update on this? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on issue #8755: Support raw index for multi-valued columns

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8755:
URL: https://github.com/apache/pinot/issues/8755#issuecomment-1328330018

   Since the topic of this issue is already supported, closing this one and use other issues to track the specific feature request


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang closed issue #8755: Support raw index for multi-valued columns

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #8755: Support raw index for multi-valued columns
URL: https://github.com/apache/pinot/issues/8755


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org