Spooling director source with compressed (zip, gzip) text files

[Phil Scala <Ph...@globalrelay.net>]

I was starting to give some thought about how to spool a set of compressed files using the spooling directory source.  At the moment I unzip them in a staging script, but maybe it would be better do that in flume.   The files are compressed using zip, gzip etc and contain 1 or more text files within.  Does anyone have any thoughts on if out of the box would be possible with flume 1.5.0.1 or higher?  I am thinking that I could need to create a custom EventDeserializer on the spooling source that would do the decompression.

Thoughts?

Thanks in advance
  Phil

Re: Spooling director source with compressed (zip, gzip) text files

[Ashish <pa...@gmail.com>]

Related JIRA https://issues.apache.org/jira/browse/FLUME-2459

Patches are welcome :)

On Mon, Aug 10, 2015 at 4:16 AM, Phil Scala <Ph...@globalrelay.net> wrote:
> I was starting to give some thought about how to spool a set of compressed
> files using the spooling directory source.  At the moment I unzip them in a
> staging script, but maybe it would be better do that in flume.   The files
> are compressed using zip, gzip etc and contain 1 or more text files within.
> Does anyone have any thoughts on if out of the box would be possible with
> flume 1.5.0.1 or higher?  I am thinking that I could need to create a custom
> EventDeserializer on the spooling source that would do the decompression.
>
>
>
> Thoughts?
>
>
>
> Thanks in advance
>
>   Phil



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal