integration with Zeppelin

[Wojciech Indyk <wo...@gmail.com>]

Hi All!
I am new here. I find NiFi as a great project for data routing. However,
extendability of NiFi give us a great potential to expand application of
NiFi routing. Recently I looked at Kylo- a software for management of data
lake (based on NiFi and its templates). In this composition of NiFi and
Kylo I feel lack of a component able to do custom data-analytics. I asked
the Kylo community on possible integration of Kylo with Zeppelin here:
https://groups.google.com/forum/#!topic/kylo-community/e6JdzneAnV0

The Kylo developers advised me to asked about possible integration here. I
haven't found any NiFi processor for Zeppelin, that enables to work on the
input from NiFi and handle the output from Zeppelin. The idea of such
integration comes from functionality I've seen in Dataiku solution. There
is a processor, that is able to run arbitrary code in the middle of
processing by specific input and output variables. Having such opportunity
the data governance in area of collecting data done by NiFi can be extended
on data governance regarding data processing, like training of Machine
Learning models, custom visualizations, etc.

What do you think of this idea? Is any NiFi processor, that supports such
integration? Is it in line with a NiFi roadmap?

--
Kind regards/ Pozdrawiam,
Wojciech Indyk

Re: integration with Zeppelin

[James Wing <jv...@gmail.com>]

I do not believe NiFi has any specific features for Zeppelin yet, but it is
possible to write custom Zeppelin code paragraphs that communicate with the
NiFi API to pull data or inspect flow status.  For an example, I recommend
Pierre Villard's US Presidential Election: tweet analysis using HDF/NiFi,
Spark, Hive and Zeppelin (
https://community.hortonworks.com/articles/30213/us-presidential-election-tweet-analysis-using-hdfn.html
).

The most typical usage is to have NiFi write to a durable store like HDFS,
HBase, SQL, etc., and then have Zeppelin read from the data store.  Would
that not work for you?

Thanks,

James

On Thu, Jun 8, 2017 at 7:09 AM, Wojciech Indyk <wo...@gmail.com>
wrote:

> Hi All!
> I am new here. I find NiFi as a great project for data routing. However,
> extendability of NiFi give us a great potential to expand application of
> NiFi routing. Recently I looked at Kylo- a software for management of data
> lake (based on NiFi and its templates). In this composition of NiFi and
> Kylo I feel lack of a component able to do custom data-analytics. I asked
> the Kylo community on possible integration of Kylo with Zeppelin here:
> https://groups.google.com/forum/#!topic/kylo-community/e6JdzneAnV0
>
> The Kylo developers advised me to asked about possible integration here. I
> haven't found any NiFi processor for Zeppelin, that enables to work on the
> input from NiFi and handle the output from Zeppelin. The idea of such
> integration comes from functionality I've seen in Dataiku solution. There
> is a processor, that is able to run arbitrary code in the middle of
> processing by specific input and output variables. Having such opportunity
> the data governance in area of collecting data done by NiFi can be extended
> on data governance regarding data processing, like training of Machine
> Learning models, custom visualizations, etc.
>
> What do you think of this idea? Is any NiFi processor, that supports such
> integration? Is it in line with a NiFi roadmap?
>
> --
> Kind regards/ Pozdrawiam,
> Wojciech Indyk
>