You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/04/16 00:17:03 UTC
[GitHub] [incubator-pinot] GSharayu opened a new issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression encoding for raw index
GSharayu opened a new issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804
When the forward index is not dictionary encoded, we have 2 choices: * store the data as is (RAW)
store the data snappy compressed - using snappy compression codec library
In addition to snappy, we should add support for other compression codecs subject to their availability in Java libraries.
Currently by default we use Snappy compression. However, this didn't really give good compression ratio for LTI's free-text data. LZO is known to provide better compression ratio and speed for larger char/varchar data.
So, we should explore other options
Firstly, we should start with simple test case to compress and uncompress direct byte buffer and do some functional and performance tests.
see ZSTD library in Java - https://github.com/luben/zstd-jni
Any new ideas/suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821728703
> Right. Since all the per column index/encoding/compressions and any tuning info is in table config, may be we can continue to have it in the config. The table config already has a field to capture this.
Got it. For this part, if users change the config for a column compression type, do we consider rebuilding the column data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821527069
I feel we should separate encoding and compression.
Maybe add two new fields into the schema.
For raw index encoding, we can support multiple options for different types:
e.g. INT/LONG : Delta/DoubleDelta/Gorrila
FLOAT/DOUBLE: Gorrila
For compression, we can support LZO, LZ4, ZSTD, DEFLATE, GZIP, SNAPPY etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821585761
> > > In columnar database world, encoding is commonly referred to column level compression techniques which play nice with columnar query execution - dictionary encoding, RLE, delta where the true benefit is that query processing can happen faster on compressed columnar data (e.g dictionary encoding) and obviously storage saving is there
> > > The purpose of this issue is to not add any new column level encoding. I was thinking of having a separate issue to enhance column level encoding support with RLE, DELTA, PFORDELTA etc.
> > > This issue is for supporting additional data compression codecs for raw data which are currently snappy compressed
> >
> >
> > Got it, so we should still make it columnar and do it at block level? Then we still need to add this to schema right?
>
> Yes this will also be columnar and will be block level although LZ4 supports some form of streaming/frame. I am not sure why do we need to add it to schema? Do you mean configuring it via table config?
Cause we want to allow tuning compression per column basis, e.g. column1 in snappy and column2 in lz4 right?
This info can be stored:
- either inside FieldSpec in schema,
- or add a new field in tableConfig, with a map of columns to compression type mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821326658
I'd also suggest Gorilla TSZ compression to the list, which was proposed by Facebook [Gorilla project](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf). This compression algorithm is adopted by Uber's [m3TSZ](https://github.com/m3db/m3/tree/master/src/dbnode/encoding/m3tsz), which showed 40% improvement over standard TSZ, [observed from Uber's production workloads](https://m3db.io/docs/architecture/m3db/engine/#m3tsz).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821576175
> > In columnar database world, encoding is commonly referred to column level compression techniques which play nice with columnar query execution - dictionary encoding, RLE, delta where the true benefit is that query processing can happen faster on compressed columnar data (e.g dictionary encoding) and obviously storage saving is there
> > The purpose of this issue is to not add any new column level encoding. I was thinking of having a separate issue to enhance column level encoding support with RLE, DELTA, PFORDELTA etc.
> > This issue is for supporting additional data compression codecs for raw data which are currently snappy compressed
>
> Got it, so we should still make it columnar and do it at block level? Then we still need to add this to schema right?
Yes this will also be columnar and will be block level although LZ4 supports some form of streaming/frame. I am not sure why do we need to add it to schema? Do you mean configuring it via table config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia edited a comment on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
siddharthteotia edited a comment on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821559734
In columnar database world, encoding is commonly referred to column level compression techniques which play nice with columnar query execution - dictionary encoding, RLE, delta where the true benefit is that query processing can happen faster on compressed columnar data (e.g dictionary encoding) and obviously storage saving is there
The purpose of this issue is to not add any new column level encoding. I was thinking of having a separate issue to enhance column level encoding support with RLE, DELTA, PFORDELTA etc.
This issue is for supporting additional data compression codecs for raw data which are currently snappy compressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821061826
there is some previous discussion here: https://github.com/apache/incubator-pinot/issues/5407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821559734
In columnar database world, encoding is commonly referred to column level compression techniques which play nice with columnar query execution - dictionary encoding, RLE, delta where the true benefit is that query processing can happen on compressed columnar data (e.g dictionary encoding)
The purpose of this issue is to not add any new column level encoding. I was thinking of having a separate issue to enhance column level encoding support with RLE, DELTA, PFORDELTA etc.
This issue is for supporting additional data compression codecs for raw data which are currently snappy compressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821588154
Right. Since all the per column index/encoding/compressions and any tuning info is in table config, may be we can continue to have it in the config. The table config already has a field to capture this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6804: Support LZO, LZ4, ZSTD, DEFLATE, GZIP compression codecs for raw index
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6804:
URL: https://github.com/apache/incubator-pinot/issues/6804#issuecomment-821570944
> In columnar database world, encoding is commonly referred to column level compression techniques which play nice with columnar query execution - dictionary encoding, RLE, delta where the true benefit is that query processing can happen faster on compressed columnar data (e.g dictionary encoding) and obviously storage saving is there
>
> The purpose of this issue is to not add any new column level encoding. I was thinking of having a separate issue to enhance column level encoding support with RLE, DELTA, PFORDELTA etc.
>
> This issue is for supporting additional data compression codecs for raw data which are currently snappy compressed
Got it, so we should still make it columnar and do it at block level? Then we still need to add this to schema right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org