[jira] [Created] (NIFI-3191) HDFS Processors Should Allow Choosing LZO Compression

["Bryan Bende (JIRA)" <ji...@apache.org>]

Bryan Bende created NIFI-3191:
---------------------------------

             Summary: HDFS Processors Should Allow Choosing LZO Compression
                 Key: NIFI-3191
                 URL: https://issues.apache.org/jira/browse/NIFI-3191
             Project: Apache NiFi
          Issue Type: Improvement
    Affects Versions: 0.7.1, 1.1.0
            Reporter: Bryan Bende


The HDFS processors have a CompressionType enum that maps the enum to a class name, currently it has the following:

{code}
case NONE: return "NONE";
case DEFAULT: return DefaultCodec.class.getName();
case BZIP: return BZip2Codec.class.getName();
case GZIP: return GzipCodec.class.getName();
case LZ4: return Lz4Codec.class.getName();
case SNAPPY: return SnappyCodec.class.getName();
case AUTOMATIC: return "Automatically Detected";
{code}

The Lzo codec class is GPL (http://code.google.com/p/hadoop-gpl-compression and https://github.com/twitter/hadoop-lzo) so it can not be shipped with NiFi or Hadoop, but if someone adds it to their Hadoop distribution there should be a way to select it from NiFi.

Seems LZO could be added to the enumeration with a mapping of:
{code}
case LZO: return "com.hadoop.compression.lzo.LzoCodec";
{code}

The libraries would still need to be linked through the LD_LIBRARY_PATH such as:
{code}
java.arg.15=-Djava.libaray.path=/path/to/your/lzocodec.so
{code}
and possibly point the additional classpath resource to the jar containing LzoCodec.java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)