You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by QiangCai <qi...@qq.com> on 2017/04/05 10:31:32 UTC

[DISCUSSION]implement delta encoding for numeric type column in SORT_COLUMNS

Hi all,

    Now we plan to implement delta encoding for the numeric type column in
SORT_COLUMNS. 
    
    1. use delta encoding to encode the numeric type data

    2. write present metadata to the page header, to record the null value.

    3. improve the compression on no-dictionary string column.
        use RLE to compress the array of the lengths in LV encoding.

    Any thoughts, comments and questions ?

Best Regards
David QiangCai



--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-implement-delta-encoding-for-numeric-type-column-in-SORT-COLUMNS-tp10124.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.

Re: [DISCUSSION]implement delta encoding for numeric type column in SORT_COLUMNS

Posted by Liang Chen <ch...@gmail.com>.
Hi David

Thanks for your starting this new feature's discussion.
Can you explain what are the major benefits after doing delta encoding for
the numeric type column.

Regards
Liang

2017-04-05 16:01 GMT+05:30 QiangCai <qi...@qq.com>:

> Hi all,
>
>     Now we plan to implement delta encoding for the numeric type column in
> SORT_COLUMNS.
>
>     1. use delta encoding to encode the numeric type data
>
>     2. write present metadata to the page header, to record the null value.
>
>     3. improve the compression on no-dictionary string column.
>         use RLE to compress the array of the lengths in LV encoding.
>
>     Any thoughts, comments and questions ?
>
> Best Regards
> David QiangCai
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> implement-delta-encoding-for-numeric-type-column-in-SORT-
> COLUMNS-tp10124.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>



-- 
Regards
Liang

Re: [DISCUSSION]implement delta encoding for numeric type column in SORT_COLUMNS

Posted by Jacky Li <ja...@qq.com>.
> 在 2017年4月5日，下午6:31，QiangCai <qi...@qq.com> 写道：
> 
> Hi all,
> 
>    Now we plan to implement delta encoding for the numeric type column in
> SORT_COLUMNS. 
> 
>    1. use delta encoding to encode the numeric type data
> 

I think the adaptive data type conversion still apply here, right?

>    2. write present metadata to the page header, to record the null value.
> 
>    3. improve the compression on no-dictionary string column.
>        use RLE to compress the array of the lengths in LV encoding.
> 

I think local dictionary is more suitable for this case

>    Any thoughts, comments and questions ?
> 
> Best Regards
> David QiangCai
> 
> 
> --
> View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-implement-delta-encoding-for-numeric-type-column-in-SORT-COLUMNS-tp10124.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.