You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Mohit Anchlia <mo...@gmail.com> on 2012/09/08 03:11:31 UTC
Debug HDFS sink
I am using HDFS sink with snappycompression. After I write data and read
from the hdfs file I only see the keys but not data. How can I debug this
with flume?
DEFINE SequenceFileLoader
org.apache.pig.piggybank.storage.SequenceFileLoader();
A = LOAD '/flume_vol/flume/2012/09/07/17/dslg1/*.snappy*' USING
SequenceFileLoader AS (key:long, document:bytearray);
DUMP A;
(1347065726395,)
(1347065726395,)
(1347065726395,)
(1347065726395,)
(1347065726395,)
(1347065726395,)
(1347065726396,)
(1347065726396,)
(1347065726396,)
(1347065726396,)
(1347065726396,)
Re: Debug HDFS sink
Posted by Alexander Lorenz <wg...@gmail.com>.
Can you sent the input to logger and pipe that through snappy to unpack them?
Another task could be to disable snappy and check the plain files if they some missing, because snappy does only compress.
regards,
Alex
On Sep 8, 2012, at 3:11 AM, Mohit Anchlia <mo...@gmail.com> wrote:
> I am using HDFS sink with snappycompression. After I write data and read
> from the hdfs file I only see the keys but not data. How can I debug this
> with flume?
>
>
> DEFINE SequenceFileLoader
> org.apache.pig.piggybank.storage.SequenceFileLoader();
>
> A = LOAD '/flume_vol/flume/2012/09/07/17/dslg1/*.snappy*' USING
> SequenceFileLoader AS (key:long, document:bytearray);
>
> DUMP A;
>
> (1347065726395,)
>
> (1347065726395,)
>
> (1347065726395,)
>
> (1347065726395,)
>
> (1347065726395,)
>
> (1347065726395,)
>
> (1347065726396,)
>
> (1347065726396,)
>
> (1347065726396,)
>
> (1347065726396,)
>
> (1347065726396,)
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF