You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Rao, Mallik" <Ma...@fmr.com> on 2013/03/19 22:11:27 UTC

Compressed Files NPE

Hello

I am trying to write compressed files in HDFS. If I am using the Flume CompressedStream type with HdfsSink, it is working fine.
But If I am writing code outside and running, it is giving null pointer exception. I must be missing something. Any help?
If I change to simple file to sequence file it is working. If I do not have compression then also it is working.

Error:
java.lang.NullPointerException
        at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:64)
        at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:97)
        at cieg.atap.DiSnAbs.open(DiSnAbs.java:52)
        at cieg.atap.DiAgManager$DiAgWorker.run(DiAgManager.java:205)

SampleCode:
        InputStream in = new FileInputStream(new File(inFile));
        OutputStream out = fileSystem.create(outFile);
        if (System.getProperty("hdfs.compression.type") != null) {
            CompressionCodec lCodec = getCodec(System.getProperty("hdfs.compression.type"));
            out = lCodec.createOutputStream(out);
        }


    public static CompressionCodec getCompressionCodec(String value)
    {   
        if (value.equals("gzip"))
            return new GzipCodec();
        else if (value.equals("bzip2"))
            return new BZip2Codec();
        else
            return new SnappyCodec();
    }

Thanks
Mallikharjuna Rao
Director, Software Engineering
603-791-8604 (W) 603 296 7721 (C)

Re: Compressed Files NPE

Posted by Mike Percy <mp...@apache.org>.
Hadoop is open source so you could look @ the line that is throwing... but I'm guessing your classpath or library path is missing something.

Regards,
Mike

Sent from my iPhone

On Mar 20, 2013, at 5:06 AM, "Rao, Mallik" <Ma...@fmr.com> wrote:

> Goal is to put inside as a custom sink. But for now testing outside FLUME.
> 
> Thanks
> Mallikharjuna Rao
> Director, Software Engineering
> 603-791-8604 (W) 603 296 7721 (C)
> 
> -----Original Message-----
> From: Mike Percy [mailto:mpercy@apache.org] 
> Sent: Tuesday, March 19, 2013 7:05 PM
> To: dev@flume.apache.org
> Subject: Re: Compressed Files NPE
> 
> Hi Mallik,
> 
> On Tue, Mar 19, 2013 at 2:11 PM, Rao, Mallik <Ma...@fmr.com>wrote:
> 
>> But If I am writing code outside and running, it is giving null pointer
>> exception.
> 
> 
> What do you mean writing code outside? I'm not sure I understand what
> you're doing here. Is this still within Flume?
> 
> Regards,
> Mike

RE: Compressed Files NPE

Posted by "Rao, Mallik" <Ma...@fmr.com>.
Goal is to put inside as a custom sink. But for now testing outside FLUME.

Thanks
Mallikharjuna Rao
Director, Software Engineering
603-791-8604 (W) 603 296 7721 (C)

-----Original Message-----
From: Mike Percy [mailto:mpercy@apache.org] 
Sent: Tuesday, March 19, 2013 7:05 PM
To: dev@flume.apache.org
Subject: Re: Compressed Files NPE

Hi Mallik,

On Tue, Mar 19, 2013 at 2:11 PM, Rao, Mallik <Ma...@fmr.com>wrote:

> But If I am writing code outside and running, it is giving null pointer
> exception.


What do you mean writing code outside? I'm not sure I understand what
you're doing here. Is this still within Flume?

Regards,
Mike

Re: Compressed Files NPE

Posted by Mike Percy <mp...@apache.org>.
Hi Mallik,

On Tue, Mar 19, 2013 at 2:11 PM, Rao, Mallik <Ma...@fmr.com>wrote:

> But If I am writing code outside and running, it is giving null pointer
> exception.


What do you mean writing code outside? I'm not sure I understand what
you're doing here. Is this still within Flume?

Regards,
Mike