You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/10/22 18:29:07 UTC
[GitHub] [pinot] richardstartin opened a new issue #7616: Raw forward index enhancement proposal
richardstartin opened a new issue #7616:
URL: https://github.com/apache/pinot/issues/7616
We would like to introduce a new format for raw forward indexes, which does not need a constant number of documents per chunk, instead partitioning columns based on uncompressed size. It is expected that this design will lead to:
* less memory consumption when there are large values in a raw column
* fewer chunks than when the number of documents is derived
* more balanced chunk sizes than when the number of documents is derived
* will provide support for realtime segments by breaking the dependency on column statistics for sizing
The format would be opt in for the foreseeable future.
[Design document](https://docs.google.com/document/d/1Y7MyQGmDD2fI7brOOFQtToxd8ML837qRuc3IlNYFvCw/edit?usp=sharing)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #7616: Raw forward index enhancement proposal
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #7616:
URL: https://github.com/apache/pinot/issues/7616#issuecomment-975796507
Implemented in #7661, can be enabled with field level raw index version config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin closed issue #7616: Raw forward index enhancement proposal
Posted by GitBox <gi...@apache.org>.
richardstartin closed issue #7616:
URL: https://github.com/apache/pinot/issues/7616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org