Re: Support Zstd as Column Compressor

[xuchuanyin <xu...@hust.edu.cn>]

As a result of the latest implementation, I store the compressor name in the
thrift and the old enum for compression_codec has been deprecated. This
makes it easier to support other compressors. Take LZ4 for example, the
following changes are required: 
1 Implement Lz4Compressor 
2 Add Lz4Compressor to the compressor factory as native supported compressor



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Support Zstd as Column Compressor

[xuchuanyin <xu...@hust.edu.cn>]

Yeah. Zstd and Snappy knows the size of decompress size from the compressed
data, but LZ4 don't. I find a link to describe this:
https://github.com/lz4/lz4-java/issues/26

To work around with LZ4, you can go with your proposal and save&use the
decompress size in the meta.
But I'd like to wrap the LZ4 implementation by
1. adding original size when we return the compressed content
2. extracting the original size when we want to decompress the content.
In this way, we can make the API stable.




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Support Zstd as Column Compressor

[xuchuanyin <xu...@hust.edu.cn>]

Snappy and Zstd both know the decompress size of the content since they stored that size along with the compressed content. But LZ4 didn't do this, you can refer to the issue#26 in the lz4-java github page.




To work around this, You can store the original size in metadata for decompression. But I would like to store the size along with the compressed content to keep the interface stable.


1. While compressing, you can return the compressed content prefixed with original size 


2. While decompressing, you can extract the original size and init the dest buffer




Please let me know your choice.

Re: Support Zstd as Column Compressor

[Jacky Li <ja...@qq.com>]

Great, thanks for your effort.

For the lz4 task,I checked lz4 compressor (lz4-java), and found it needs the decompressed size before decompressing the data. In CarbonData V3 format, we have stored the uncompressed size of data page in ChunkCompressionMeta.total_uncompressed_size in the data file for every page. 
So to implement Lz4Compressor in carbon, I think we need to use this information in the file and maybe compressor interface need to changed to add this parameter to unCompressXXX interface so that lz4 can use it.

Regards,
Jacky

> 在 2018年9月12日，下午8:35，xuchuanyin <xu...@hust.edu.cn> 写道：
> 
> As a result of the latest implementation, I store the compressor name in the
> thrift and the old enum for compression_codec has been deprecated. This
> makes it easier to support other compressors. Take LZ4 for example, the
> following changes are required: 
> 1 Implement Lz4Compressor 
> 2 Add Lz4Compressor to the compressor factory as native supported compressor
> 
> 
> --
> Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>