[jira] [Moved] (HADOOP-19167) Change of Codec configuration does not work

["Yiqun Lin (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yiqun Lin moved HDFS-17510 to HADOOP-19167:
-------------------------------------------

    Component/s: compress
                     (was: compress)
            Key: HADOOP-19167  (was: HDFS-17510)
        Project: Hadoop Common  (was: Hadoop HDFS)

> Change of Codec configuration does not work
> -------------------------------------------
>
>                 Key: HADOOP-19167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19167
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: compress
>            Reporter: Zhikai Hu
>            Priority: Minor
>              Labels: pull-request-available
>
> In one of my projects, I need to dynamically adjust compression level for different files. 
> However, I found that in most cases the new compression level does not take effect as expected, the old compression level continues to be used.
> Here is the relevant code snippet:
> ZStandardCodec zStandardCodec = new ZStandardCodec();
> zStandardCodec.setConf(conf);
> conf.set("io.compression.codec.zstd.level", "5"); // level may change dynamically
> conf.set("io.compression.codec.zstd", zStandardCodec.getClass().getName());
> writer = SequenceFile.createWriter(conf, SequenceFile.Writer.file(sequenceFilePath),
>                                 SequenceFile.Writer.keyClass(LongWritable.class),
>                                 SequenceFile.Writer.valueClass(BytesWritable.class),
>                                 SequenceFile.Writer.compression(CompressionType.BLOCK));
> The reason is SequenceFile.Writer.init() method will call CodecPool.getCompressor(codec, null) to get a compressor. 
> If the compressor is a reused instance, the conf is not applied because it is passed as null:
> public static Compressor getCompressor(CompressionCodec codec, Configuration conf) {
> Compressor compressor = borrow(compressorPool, codec.getCompressorType());
> if (compressor == null)
> { compressor = codec.createCompressor(); LOG.info("Got brand-new compressor ["+codec.getDefaultExtension()+"]"); }
> else {
> compressor.reinit(conf);   //conf is null here
> ......
>  
> Please also refer to my unit test to reproduce the bug. 
> To address this bug, I modified the code to ensure that the configuration is read back from the codec when a compressor is reused.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org