You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Banias H <ba...@gmail.com> on 2017/04/19 21:43:08 UTC

Decompressing content consumed from Kafka topics?

Hi,

I have NiFi 1.1.1 and would like to compress messages before sending to a
Kafka topic.
 I can do this in the PublishKafka_0_10 processor. There is a "Compression
Type" property where I can choose:

- none
- gzip
- snappy
- lz4

But in the ConsumeKafka_0_10 processor, there is no option to decompress
messages. *Is there a way to decompress messages from Kafka topics in the
consumer flow?*

I tried the following setup:

*Producer Flow*
- In PublishKafka_0_10 processor, set "Compression Type" to either gzip or
snappy

*Consumer Flow*
- In Compress processor, set "Mode" set to "decompress" and set
"Compression Format" to either gzip or snappy. However both failed to
decompress.

- Tried UnpackContent processor, but there is no "gzip", "snappy", or even
"lz4" I can select.

If anyone has a suggestion on how to decompress Kafka messages that are
compressed by PublishKafka_0_10 processor, I would really appreciate it.

Thanks,
BH

Re: Decompressing content consumed from Kafka topics?

Posted by Banias H <ba...@gmail.com>.

Thanks Joe. It does look like a misconfiguration on my end. I started with
a new topic and everything works fine.

On Wed, Apr 19, 2017 at 4:45 PM, Joe Witt <jo...@gmail.com> wrote:

> Hello
>
> UnpackContent is for archive formats.  gzip/snappy/lz4 aren't archive
> formats but are compression types.  You want CompressContent in
> decompress mode.  I think the first question is why they didn't
> decompress.  Did you verify the were indeed still compressed?  You can
> use IdentifyMimeType to do a quick check of what it thinks they are.
> Or grab the content from nifi's provenance capability and look at it
> using linux tools.
>
> Thanks
> Joe
>
> On Wed, Apr 19, 2017 at 5:43 PM, Banias H <ba...@gmail.com> wrote:
> > Hi,
> >
> > I have NiFi 1.1.1 and would like to compress messages before sending to a
> > Kafka topic.
> >  I can do this in the PublishKafka_0_10 processor. There is a
> "Compression
> > Type" property where I can choose:
> >
> > - none
> > - gzip
> > - snappy
> > - lz4
> >
> > But in the ConsumeKafka_0_10 processor, there is no option to decompress
> > messages. Is there a way to decompress messages from Kafka topics in the
> > consumer flow?
> >
> > I tried the following setup:
> >
> > Producer Flow
> > - In PublishKafka_0_10 processor, set "Compression Type" to either gzip
> or
> > snappy
> >
> > Consumer Flow
> > - In Compress processor, set "Mode" set to "decompress" and set
> "Compression
> > Format" to either gzip or snappy. However both failed to decompress.
> >
> > - Tried UnpackContent processor, but there is no "gzip", "snappy", or
> even
> > "lz4" I can select.
> >
> > If anyone has a suggestion on how to decompress Kafka messages that are
> > compressed by PublishKafka_0_10 processor, I would really appreciate it.
> >
> > Thanks,
> > BH
> >
> >
>

Re: Decompressing content consumed from Kafka topics?

Posted by Joe Witt <jo...@gmail.com>.

Hello

UnpackContent is for archive formats.  gzip/snappy/lz4 aren't archive
formats but are compression types.  You want CompressContent in
decompress mode.  I think the first question is why they didn't
decompress.  Did you verify the were indeed still compressed?  You can
use IdentifyMimeType to do a quick check of what it thinks they are.
Or grab the content from nifi's provenance capability and look at it
using linux tools.

Thanks
Joe

On Wed, Apr 19, 2017 at 5:43 PM, Banias H <ba...@gmail.com> wrote:
> Hi,
>
> I have NiFi 1.1.1 and would like to compress messages before sending to a
> Kafka topic.
>  I can do this in the PublishKafka_0_10 processor. There is a "Compression
> Type" property where I can choose:
>
> - none
> - gzip
> - snappy
> - lz4
>
> But in the ConsumeKafka_0_10 processor, there is no option to decompress
> messages. Is there a way to decompress messages from Kafka topics in the
> consumer flow?
>
> I tried the following setup:
>
> Producer Flow
> - In PublishKafka_0_10 processor, set "Compression Type" to either gzip or
> snappy
>
> Consumer Flow
> - In Compress processor, set "Mode" set to "decompress" and set "Compression
> Format" to either gzip or snappy. However both failed to decompress.
>
> - Tried UnpackContent processor, but there is no "gzip", "snappy", or even
> "lz4" I can select.
>
> If anyone has a suggestion on how to decompress Kafka messages that are
> compressed by PublishKafka_0_10 processor, I would really appreciate it.
>
> Thanks,
> BH
>
>