You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nifi.apache.org by Chris Herssens <ch...@gmail.com> on 2017/08/04 06:22:17 UTC

using avro flume source in nifi

Hello All,

I want to configure a flume source processor using avro as flume source.
When I start the ExecuteFlumeSource, I don't get any errors, but I can't
connect to the source port.

telnet localhost 5151 gives connection refused.

The configuration is as follows

type = avro

agent name = a1

source name = batchSource

Flume configuration

   a1.sources.batchSource.type = avro

   a1.sources.batchSource.bind = 0.0.0.0

   a1.sources.batchSource.port=5151


Using netcat as flume source works.

Re: using avro flume source in nifi

Posted by Chris Herssens <ch...@gmail.com>.

Hello Joe,

Currently we use Flume for sending avro message to our big data platform.
Now I want to replace flume with NIfi. Since the sender is a  flume agent,
I need to use the flume source.
If I configure this sourcs for receiving netcat, I can access the tcp port.
Using avro as type doesn't work. If I set "run schedule" ti 60 sec, then it
work, I can telnet to the configured tcp port.

Regards,

Chris

On Fri, Aug 11, 2017 at 4:03 AM, Joe Witt <jo...@gmail.com> wrote:

> Chris,
>
> I dont believe there are many in the community here familiar enough
> with Flume so that might make getting help here a bit tougher.  Are
> you wanting NiFi to act as a TCP socket listener on a given port
> receiving Avro objects?
>
> Thanks
>
> On Thu, Aug 3, 2017 at 11:22 PM, Chris Herssens
> <ch...@gmail.com> wrote:
> > Hello All,
> >
> > I want to configure a flume source processor using avro as flume source.
> > When I start the ExecuteFlumeSource, I don't get any errors, but I can't
> > connect to the source port.
> >
> > telnet localhost 5151 gives connection refused.
> >
> > The configuration is as follows
> >
> > type = avro
> >
> > agent name = a1
> >
> > source name = batchSource
> >
> > Flume configuration
> >
> >    a1.sources.batchSource.type = avro
> >
> >    a1.sources.batchSource.bind = 0.0.0.0
> >
> >    a1.sources.batchSource.port=5151
> >
> >
> > Using netcat as flume source works.
>

Re: using avro flume source in nifi

Posted by Joe Witt <jo...@gmail.com>.

Chris,

I dont believe there are many in the community here familiar enough
with Flume so that might make getting help here a bit tougher.  Are
you wanting NiFi to act as a TCP socket listener on a given port
receiving Avro objects?

Thanks

On Thu, Aug 3, 2017 at 11:22 PM, Chris Herssens
<ch...@gmail.com> wrote:
> Hello All,
>
> I want to configure a flume source processor using avro as flume source.
> When I start the ExecuteFlumeSource, I don't get any errors, but I can't
> connect to the source port.
>
> telnet localhost 5151 gives connection refused.
>
> The configuration is as follows
>
> type = avro
>
> agent name = a1
>
> source name = batchSource
>
> Flume configuration
>
>    a1.sources.batchSource.type = avro
>
>    a1.sources.batchSource.bind = 0.0.0.0
>
>    a1.sources.batchSource.port=5151
>
>
> Using netcat as flume source works.