You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by kishore g <g....@gmail.com> on 2020/02/11 22:10:03 UTC
Convert dictionary encoded into raw
As of today, we apply dictionary encoding for all columns by default. We
should probably move a hybrid approach where we decide the encoding based
on the data profile. For e.g. if the cardinality of the column is very high
(which is the case for metrics), dictionary encoding does not provide a lot
of value.
Thoughts?
Re: Convert dictionary encoded into raw
Posted by Mayank Shrivastava <ma...@gmail.com>.
+1, we may also do this automatically
Sent from my iPhone
> On Feb 11, 2020, at 2:10 PM, kishore g <g....@gmail.com> wrote:
>
> As of today, we apply dictionary encoding for all columns by default. We
> should probably move a hybrid approach where we decide the encoding based
> on the data profile. For e.g. if the cardinality of the column is very high
> (which is the case for metrics), dictionary encoding does not provide a lot
> of value.
>
> Thoughts?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
For additional commands, e-mail: dev-help@pinot.apache.org
Re: Convert dictionary encoded into raw
Posted by Mayank Shrivastava <ma...@gmail.com>.
I think I shares the formula for threshold long time ago. Let me dig it up and share.
Sent from my iPhone
> On Feb 11, 2020, at 2:33 PM, kishore g <g....@gmail.com> wrote:
>
> Any idea on what should be the threshold?
>
> Good point about the variable-length column, should we rely on a bloom
> filter instead?
>
>> On Tue, Feb 11, 2020 at 2:14 PM Siddharth Teotia
>> <st...@linkedin.com.invalid> wrote:
>>
>> Yes. Especially if the data is fixed width and high cardinality then
>> dictionary encoding is not going to be very useful.
>>
>> May be for fixed width, we should create dictionary only if cardinality is
>> below a certain threshold?
>>
>> For variable width, whether cardinality is high or low, dictionary
>> encoding will improve filter processing if column is used heavily in
>> filters. So may be for variable width we should always create dictionary
>> unless indicated otherwise in table config?
>> ________________________________
>> From: kishore g <g....@gmail.com>
>> Sent: Tuesday, February 11, 2020 2:10 PM
>> To: dev@pinot.apache.org <de...@pinot.apache.org>
>> Subject: Convert dictionary encoded into raw
>>
>> As of today, we apply dictionary encoding for all columns by default. We
>> should probably move a hybrid approach where we decide the encoding based
>> on the data profile. For e.g. if the cardinality of the column is very high
>> (which is the case for metrics), dictionary encoding does not provide a lot
>> of value.
>>
>> Thoughts?
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
For additional commands, e-mail: dev-help@pinot.apache.org
Re: Convert dictionary encoded into raw
Posted by Subbu Subramaniam <mc...@apache.org>.
The threshold should probably be set as a percentage of the number of rows in the segment.
Also, we should implement this by setting the encoding to be "auto" or something like that. That way, we can expect that some segments may have dictionary encoding and others may not. Otherwise, there will be inconsistency between table config and what exists in (some) segments
-Subbu
On 2020/02/11 22:33:34, kishore g <g....@gmail.com> wrote:
> Any idea on what should be the threshold?
>
> Good point about the variable-length column, should we rely on a bloom
> filter instead?
>
> On Tue, Feb 11, 2020 at 2:14 PM Siddharth Teotia
> <st...@linkedin.com.invalid> wrote:
>
> > Yes. Especially if the data is fixed width and high cardinality then
> > dictionary encoding is not going to be very useful.
> >
> > May be for fixed width, we should create dictionary only if cardinality is
> > below a certain threshold?
> >
> > For variable width, whether cardinality is high or low, dictionary
> > encoding will improve filter processing if column is used heavily in
> > filters. So may be for variable width we should always create dictionary
> > unless indicated otherwise in table config?
> > ________________________________
> > From: kishore g <g....@gmail.com>
> > Sent: Tuesday, February 11, 2020 2:10 PM
> > To: dev@pinot.apache.org <de...@pinot.apache.org>
> > Subject: Convert dictionary encoded into raw
> >
> > As of today, we apply dictionary encoding for all columns by default. We
> > should probably move a hybrid approach where we decide the encoding based
> > on the data profile. For e.g. if the cardinality of the column is very high
> > (which is the case for metrics), dictionary encoding does not provide a lot
> > of value.
> >
> > Thoughts?
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
For additional commands, e-mail: dev-help@pinot.apache.org
Re: Convert dictionary encoded into raw
Posted by kishore g <g....@gmail.com>.
Any idea on what should be the threshold?
Good point about the variable-length column, should we rely on a bloom
filter instead?
On Tue, Feb 11, 2020 at 2:14 PM Siddharth Teotia
<st...@linkedin.com.invalid> wrote:
> Yes. Especially if the data is fixed width and high cardinality then
> dictionary encoding is not going to be very useful.
>
> May be for fixed width, we should create dictionary only if cardinality is
> below a certain threshold?
>
> For variable width, whether cardinality is high or low, dictionary
> encoding will improve filter processing if column is used heavily in
> filters. So may be for variable width we should always create dictionary
> unless indicated otherwise in table config?
> ________________________________
> From: kishore g <g....@gmail.com>
> Sent: Tuesday, February 11, 2020 2:10 PM
> To: dev@pinot.apache.org <de...@pinot.apache.org>
> Subject: Convert dictionary encoded into raw
>
> As of today, we apply dictionary encoding for all columns by default. We
> should probably move a hybrid approach where we decide the encoding based
> on the data profile. For e.g. if the cardinality of the column is very high
> (which is the case for metrics), dictionary encoding does not provide a lot
> of value.
>
> Thoughts?
>
Re: Convert dictionary encoded into raw
Posted by Siddharth Teotia <st...@linkedin.com.INVALID>.
Yes. Especially if the data is fixed width and high cardinality then dictionary encoding is not going to be very useful.
May be for fixed width, we should create dictionary only if cardinality is below a certain threshold?
For variable width, whether cardinality is high or low, dictionary encoding will improve filter processing if column is used heavily in filters. So may be for variable width we should always create dictionary unless indicated otherwise in table config?
________________________________
From: kishore g <g....@gmail.com>
Sent: Tuesday, February 11, 2020 2:10 PM
To: dev@pinot.apache.org <de...@pinot.apache.org>
Subject: Convert dictionary encoded into raw
As of today, we apply dictionary encoding for all columns by default. We
should probably move a hybrid approach where we decide the encoding based
on the data profile. For e.g. if the cardinality of the column is very high
(which is the case for metrics), dictionary encoding does not provide a lot
of value.
Thoughts?