Getting started

[Eric Desjardins <de...@gmail.com>]

 Hi,

I am evaluating nifi to replace a local ETL system. I went through the
video and the doc and it looks great.

We have a large number of data feeds that ends up in different folder on a
sftp server. I could use a SFTP processor to fetch data from the server.
Than, I could write my own processor to process the data. For this, I would
need to know information about the feed to fetch a configuration. How would
I do that?

Than, I am curious about monitoring all those feeds. I need to know what`s
going on but also if data arrive on schedule.

How would I setup my flow?

Kind Regards,
Eric

Re: Getting started

[Ed B <bd...@gmail.com>]

Hi Eric,
For monitoring, you have following tools:
1. Monitoring in web UI (
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#monitoring ,
https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#routing-and-mediation
)
2. Using reporting tasks (
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks
 )

As for processing the data: NIFI is not ETL tool. it's a data FLOW
management tool. Yes, you have a lot of processors that can extract data
and process, make decisions, fork flows based on different attributes or
content, etc. But usually your NIFI clusters are tiny, and ETL tools a
sitting on much bigger and powerful clusters.
But, you obviously can do ETL too, Writing processors explained number of
time (e.g.
http://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/,
https://community.hortonworks.com/articles/4318/build-custom-nifi-processor.html
).

Regards,
Ed.

On Thu, Jul 19, 2018 at 8:59 AM Eric Desjardins <de...@gmail.com> wrote:

> Hi,
>
> I am evaluating nifi to replace a local ETL system. I went through the
> video and the doc and it looks great.
>
> We have a large number of data feeds that ends up in different folder on a
> sftp server. I could use a SFTP processor to fetch data from the server.
> Than, I could write my own processor to process the data. For this, I would
> need to know information about the feed to fetch a configuration. How would
> I do that?
>
> Than, I am curious about monitoring all those feeds. I need to know what`s
> going on but also if data arrive on schedule.
>
> How would I setup my flow?
>
> Kind Regards,
> Eric
>
>