[jira] [Created] (NIFI-10309) LZ4 and SNAPPY codecs doesn't work for "getter" processors for HDFS

["Katerina Navara (Jira)" <ji...@apache.org>]

Katerina Navara created NIFI-10309:
--------------------------------------

             Summary: LZ4 and SNAPPY codecs doesn't work for "getter" processors for HDFS
                 Key: NIFI-10309
                 URL: https://issues.apache.org/jira/browse/NIFI-10309
             Project: Apache NiFi
          Issue Type: Bug
    Affects Versions: 1.16.3
         Environment: CentOS Linux release 8.3.2011
Hadoop 3.3.3
Nifi 1.16.3
            Reporter: Katerina Navara


# create in HDFS some file with compression (lz4 or snappy)

 # Create GetHDFS Or FetchHDFS processor

 # set prop “{*}Compression codec{*}” to LZ4 or SNAPPY.

 # start processor

Actual result:
snappy case: 
{code:java}
"ERROR [Timer-Driven Process Thread-1] o.apache.nifi.processors.hadoop.GetHDFS GetHDFS[id=0182100b-a13c-1f5a-f526-933b6e2767a1] Error retrieving file hdfs://localhost:9000/test/snappy/file.snappy from HDFS due to org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.hadoop.io.compress.BlockDecompressorStream@5a5668ea for StandardFlowFileRecord[uuid=fe485e3e-6594-49b5-b46d-13fabbe734dd,claim=,offset=0,name=fe485e3e-6594-49b5-b46d-13fabbe734dd,size=0] due to java.lang.OutOfMemoryError: Java heap space
org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.hadoop.io.compress.BlockDecompressorStream@5a5668ea for StandardFlowFileRecord[uuid=fe485e3e-6594-49b5-b46d-13fabbe734dd,claim=,offset=0,name=fe485e3e-6594-49b5-b46d-13fabbe734dd,size=0] due to java.lang.OutOfMemoryError: Java heap space
	at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3562)
	at org.apache.nifi.processors.hadoop.GetHDFS.processBatchOfFiles(GetHDFS.java:382)"
{code}


lz4 case: 

{code:java}
o.apache.nifi.processors.hadoop.GetHDFS GetHDFS[id=01821007-a13c-1f5a-e2aa-fbea28643aac] Error retrieving file hdfs://localhost:9000/test/lz4/file.lz4 from HDFS due to java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory
{code}





--
This message was sent by Atlassian Jira
(v8.20.10#820010)