[jira] [Commented] (PIG-3681) NullpointerException while processing files in gzip format

["Kumar Ravi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877978#comment-13877978 ] 

Kumar Ravi commented on PIG-3681:
---------------------------------

I need to correct my statement: An OutofMemoryError is seen while using the textinputformat.record.delimiter property not a NullPointerException.

> NullpointerException while processing files in gzip format
> ----------------------------------------------------------
>
>                 Key: PIG-3681
>                 URL: https://issues.apache.org/jira/browse/PIG-3681
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.11
>         Environment: Linux CentOS 6.4; CDH-4.x and CDH-5.0 (beta)
>            Reporter: Kumar Ravi
>
> When pig processes a large gzip file with text or mixed text and binary content, it throws a NullPointerException if the property texinputformat.record.delimiter is set to '\n'. This is because pig interprets the specified delimiter as a two character string "\" followed by "n" and not as a new line character.
>  If this property is not set, same file unzips without problems, but the diff output of file unzipped using pig and unzipped using the gunzip command differs.
>  Steps to recreate:
> 1. create a text file that is ~ 4GB - I concatanated some pig/hadoop stdout and syslog files to create this file about 4GB in size.
> 2. compress it on unix command line - Ex. gzip abc
> 3. upload to hdfs (optional)
> 4. run the pig script included below to read/write the file.
> pig --param job_name="gunzip abc" --param inputfile="abc.gz" --param outputdir=./test --param outputfile=abc gunzip.pig
>  
> Here are the contents of gunzip.pig:
> set job.name '$job_name' 
> set textinputformat.record.delimiter "\n"; 
> gzdata = LOAD '$inputfile' USING PigStorage(); 
> STORE gzdata INTO '$outputdir/$outputfile' USING PigStorage();
> This will cause the NullPointerException.
> If the second line (set textinputformat.record.delimiter field) is commented out, the Exception won't occur but the output is not the same as the one produced by gunzip.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)