First Instance of Nifi

[Kan Wong <ka...@gmail.com>]

Hi Nifi team,

My company which is a bank in asia looking for a data integration solution
to collect all distributed data across fragmented window and linux servers
environment into a centralized data warehouse.

We just came across Nifi and Minifi which seem to be capable for the
integration and see what collaboration that we could come up with. I'm
unable to reveal the name yet but very soon..

Here the data we would like to consolidate from distributed environments.

- Order, Trade, Execution Messages across different systems (File and
database Record (MSSSQL and Oracle))
- Audit Logs file (flat file and database record (MSSQL and Oracle)
- Real time market database stream from third party via TCP
- Fragmented historical data (flat file, multiple database instances in
MSSQL)

We would like to consolidate all into a single data warehouse and integrate
all for ongoing collection.

is Nifi will fit the application?

Kan

Re: First Instance of Nifi

[Emanuel Oliveira <em...@gmail.com>]

About tcp listener, its true there's a nifi listener, but we step back as
its not 100% a resilient solution.. we wsnted to implement in nifi a
listener for jira webhook events which are sent only once, so 0% margin
loosing msgs.
I wonder what nifi more seniors experienced advise for this type of
scenario?

Some options in aws we talked:
- AWS API Gateway as tcp listener invoking lambda function to put msg on a
persistent queue (ex aws sqs, kinesis..).
- custom EC2 machine.

In reality theres no other direct tcp listener/servers in aws .. all aws
services always need a client using an api/lib to put msgs on a service..
We found this hilarious, when the perception from all jargon online seems
all options exists, all type of services exists and.. well one find out
"nope no such thing as a simpler tcp server for you".
Taking in account theres basically 2 data flows patterns "push" and "pull",
seems nifi at current state is mostly a pull only.. so for push use case
one must struggle using additional service, and then find out at least in
aws theres really no lots of options as like i mentioned above (and im
challenging here a bit, hoping im wrong) one must always implement a
listener manually uding a lib/api to then be able to interact with services
like putmsg() on aws sqs queue.. its not possible to have an aws sqs  queue
directly listening.

Thanks,
Emanuel O.


On Sat 15 Feb 2020, 19:47 Mike Thomsen, <mi...@gmail.com> wrote:

> Yes, very much so. The program I'm on uses NiFi for everything from
> processing one off Excel spreadsheets, to database records (MySQL,
> Postgres, Oracle, etc.), to log files. We also have some custom use cases I
> can't go into that are very much off the beaten path and it still excels at
> its job.
>
> With regard to your third use case, there's TCP listening support, but you
> might want to consider creating a simple TCP listener that pushes the data
> into Apache Kafka. The reason for that is because Kafka is really good for
> organizing and routing events going into a system like NiFi.
>
> Thanks,
>
> Mike
>
> On Fri, Feb 14, 2020 at 7:36 AM Kan Wong <ka...@gmail.com> wrote:
>
> > Hi Nifi team,
> >
> > My company which is a bank in asia looking for a data integration
> solution
> > to collect all distributed data across fragmented window and linux
> servers
> > environment into a centralized data warehouse.
> >
> > We just came across Nifi and Minifi which seem to be capable for the
> > integration and see what collaboration that we could come up with. I'm
> > unable to reveal the name yet but very soon..
> >
> > Here the data we would like to consolidate from distributed environments.
> >
> > - Order, Trade, Execution Messages across different systems (File and
> > database Record (MSSSQL and Oracle))
> > - Audit Logs file (flat file and database record (MSSQL and Oracle)
> > - Real time market database stream from third party via TCP
> > - Fragmented historical data (flat file, multiple database instances in
> > MSSQL)
> >
> > We would like to consolidate all into a single data warehouse and
> integrate
> > all for ongoing collection.
> >
> > is Nifi will fit the application?
> >
> > Kan
> >
>

Re: First Instance of Nifi

[Mike Thomsen <mi...@gmail.com>]

Yes, very much so. The program I'm on uses NiFi for everything from
processing one off Excel spreadsheets, to database records (MySQL,
Postgres, Oracle, etc.), to log files. We also have some custom use cases I
can't go into that are very much off the beaten path and it still excels at
its job.

With regard to your third use case, there's TCP listening support, but you
might want to consider creating a simple TCP listener that pushes the data
into Apache Kafka. The reason for that is because Kafka is really good for
organizing and routing events going into a system like NiFi.

Thanks,

Mike

On Fri, Feb 14, 2020 at 7:36 AM Kan Wong <ka...@gmail.com> wrote:

> Hi Nifi team,
>
> My company which is a bank in asia looking for a data integration solution
> to collect all distributed data across fragmented window and linux servers
> environment into a centralized data warehouse.
>
> We just came across Nifi and Minifi which seem to be capable for the
> integration and see what collaboration that we could come up with. I'm
> unable to reveal the name yet but very soon..
>
> Here the data we would like to consolidate from distributed environments.
>
> - Order, Trade, Execution Messages across different systems (File and
> database Record (MSSSQL and Oracle))
> - Audit Logs file (flat file and database record (MSSQL and Oracle)
> - Real time market database stream from third party via TCP
> - Fragmented historical data (flat file, multiple database instances in
> MSSQL)
>
> We would like to consolidate all into a single data warehouse and integrate
> all for ongoing collection.
>
> is Nifi will fit the application?
>
> Kan
>