Nifi as unprocessed UDP packet forwarder?

[Stephen Greszczyszyn <sg...@gmail.com>]

Hi there,

I am planning on using NiFi for various types of data ingestion, but was
wondering if I could use NiFi for raw UDP packet forwarding.

The data I am trying to route is:

1) Collectd (UDP): I don't think that there is a collectd processor for
NiFi, so I'd like to take the raw UDP flows and direct them to Telegraf and
Logstash (for testing purposes) that are listening locally on different
ports.


2) Syslog (UDP): I'd like to test sending raw UDP packets to multiple
destinations as well as processing/filtering syslog with the NiFi syslog
module.


3) Netflow (UDP):  I'm going to performance test receiving a large raw
netflow stream and would like to be able to filter out only the netflow
packets for the protocol types I'm interested in for my application.


So these are my use cases for now, but I'm a bit lost in the NiFi
documentation on how to route raw UDP packets without any specific
processing being done to those packets.


Thanks!

Re: Nifi as unprocessed UDP packet forwarder?

[Pierre Villard <pi...@gmail.com>]

Hi Stephen,

You probably want to use ListenUDP processor and define a listening port on
the NiFi node(s) that will receive the UDP packets. If you just want to
forward the packet as-is, then you'd possibly use PutUDP to send the UDP
packets to another destination.

In addition to that, if you want to do some processing/filtering/routing,
you can probably have a look at ListenSyslog (to automatically extract the
syslog related attributes), ListenUDPRecord to convert the raw data into
another format, and/or ParseNetflowv5 (to be released with NiFi 1.8.0).

Thanks,
Pierre

Le mer. 24 oct. 2018 à 16:21, Stephen Greszczyszyn <sg...@gmail.com> a
écrit :

> Hi there,
>
> I am planning on using NiFi for various types of data ingestion, but was
> wondering if I could use NiFi for raw UDP packet forwarding.
>
> The data I am trying to route is:
>
> 1) Collectd (UDP): I don't think that there is a collectd processor for
> NiFi, so I'd like to take the raw UDP flows and direct them to Telegraf and
> Logstash (for testing purposes) that are listening locally on different
> ports.
>
>
> 2) Syslog (UDP): I'd like to test sending raw UDP packets to multiple
> destinations as well as processing/filtering syslog with the NiFi syslog
> module.
>
>
> 3) Netflow (UDP):  I'm going to performance test receiving a large raw
> netflow stream and would like to be able to filter out only the netflow
> packets for the protocol types I'm interested in for my application.
>
>
> So these are my use cases for now, but I'm a bit lost in the NiFi
> documentation on how to route raw UDP packets without any specific
> processing being done to those packets.
>
>
> Thanks!
>