ConsumeKafka Message Demarcator

[Edgardo Vega <ed...@gmail.com>]

I have been trying to use ConsumeKafka processor. No matter what I try with
the message demarcator I can't get it to give me just one message at a time
instead it batches a bunch together.  I tried \n, shift enter, and copied
the line ending without success.

I am able to use SplitText right afterward and that does work so I have no
idea what I am doing wrong. Any help would be greatly appreciated.

-- 
Cheers,

Edgardo

Re: ConsumeKafka Message Demarcator

[Edgardo Vega <ed...@gmail.com>]

Joe,

Thanks that does work.

Do you happen to know to consume from
the __consumer_offsets topic correctly? In the kafka-console-consumer you
would use

--formatter
"kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"

I don't think there is a way to do that on the GetKafka/ConsumeKafka
processors unless I am missing something.

Cheers,

Edgardo



On Sat, Nov 19, 2016 at 9:31 PM, Joe Witt <jo...@gmail.com> wrote:

> Edgardo
>
> If you do not use the message demarcator at all then you should get a
> single message out of the topic as a single flow file each time.  The
> demarcator should be used only when you want multiple messages bundled
> together within a single flow file.  The batching/bundling enables
> higher throughput but single event at a time can be simpler for some
> use cases.
>
> Thanks
> Joe
>
> On Sat, Nov 19, 2016 at 8:43 PM, Edgardo Vega <ed...@gmail.com>
> wrote:
> > I have been trying to use ConsumeKafka processor. No matter what I try
> with
> > the message demarcator I can't get it to give me just one message at a
> time
> > instead it batches a bunch together.  I tried \n, shift enter, and copied
> > the line ending without success.
> >
> > I am able to use SplitText right afterward and that does work so I have
> no
> > idea what I am doing wrong. Any help would be greatly appreciated.
> >
> > --
> > Cheers,
> >
> > Edgardo
>



-- 
Cheers,

Edgardo

Re: ConsumeKafka Message Demarcator

[Joe Witt <jo...@gmail.com>]

Edgardo

If you do not use the message demarcator at all then you should get a
single message out of the topic as a single flow file each time.  The
demarcator should be used only when you want multiple messages bundled
together within a single flow file.  The batching/bundling enables
higher throughput but single event at a time can be simpler for some
use cases.

Thanks
Joe

On Sat, Nov 19, 2016 at 8:43 PM, Edgardo Vega <ed...@gmail.com> wrote:
> I have been trying to use ConsumeKafka processor. No matter what I try with
> the message demarcator I can't get it to give me just one message at a time
> instead it batches a bunch together.  I tried \n, shift enter, and copied
> the line ending without success.
>
> I am able to use SplitText right afterward and that does work so I have no
> idea what I am doing wrong. Any help would be greatly appreciated.
>
> --
> Cheers,
>
> Edgardo