You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Andrés Ivaldi <ia...@gmail.com> on 2018/05/11 13:53:30 UTC
First steps with Nifi
Hello, I'm new with Nifi,
I'm working on a POC where flow must be created dinamically and result
processed with spark, which would be the best way to start?
I've never used Nifi before.
Regards,
--
Ing. Ivaldi Andres
Re: First steps with Nifi
Posted by Mike Thomsen <mi...@gmail.com>.
We might want to even go so far as to roll our own based around using YARN
APIs to submit something so it's not tied to one framework.
On Fri, May 11, 2018 at 10:57 AM Joe Witt <jo...@gmail.com> wrote:
> Mike
>
> That could be cool - though we'd probably first want to reach out to
> the Kylo community and ask them to contribute it directly rather than
> just grabbing a copy.
>
> Thanks
>
> On Fri, May 11, 2018 at 10:56 AM, Damian Czaja <da...@nokia.com>
> wrote:
> > Here is a great repo with a lot of information about NiFi and tools
> around
> > it created by the community:
> > https://github.com/jfrazee/awesome-nifi
> >
> > There is also some processor to submit Spark Jobs:
> > https://github.com/qiansl127/nifi-spark-bundle
> >
> > Best regards,
> > Damian
> >
> >
> > On 11.05.2018 16:48, Mike Thomsen wrote:
> >
> > I was poking around the Kylo repo and noticed that they appear to have a
> > processor that is designed to directly launch a Spark job from NiFi (ie
> not
> > just work w/ Livy). Any thoughts on reviewing and possibly bringing that
> > into the core bundles since IIRC it's ASL 2.0?
> >
> > On Fri, May 11, 2018 at 10:27 AM dan young <da...@gmail.com> wrote:
> >>
> >> Awesome! NiFi is an amazing tool. As Joe mentioned the docs are a great
> >> place to start along with the Hortonworks forums. Although we don't use
> HDP,
> >> the content is still very relevant.
> >>
> >> Start simple and iterate...Install on your laptop and start moving bits!
> >>
> >> Regards
> >>
> >> Dano
> >>
> >>
> >> On Fri, May 11, 2018, 7:54 AM Andrés Ivaldi <ia...@gmail.com> wrote:
> >>>
> >>> Hello, I'm new with Nifi,
> >>>
> >>> I'm working on a POC where flow must be created dinamically and result
> >>> processed with spark, which would be the best way to start?
> >>>
> >>> I've never used Nifi before.
> >>>
> >>> Regards,
> >>>
> >>>
> >>> --
> >>> Ing. Ivaldi Andres
> >
> >
>
Re: First steps with Nifi
Posted by Joe Witt <jo...@gmail.com>.
Mike
That could be cool - though we'd probably first want to reach out to
the Kylo community and ask them to contribute it directly rather than
just grabbing a copy.
Thanks
On Fri, May 11, 2018 at 10:56 AM, Damian Czaja <da...@nokia.com> wrote:
> Here is a great repo with a lot of information about NiFi and tools around
> it created by the community:
> https://github.com/jfrazee/awesome-nifi
>
> There is also some processor to submit Spark Jobs:
> https://github.com/qiansl127/nifi-spark-bundle
>
> Best regards,
> Damian
>
>
> On 11.05.2018 16:48, Mike Thomsen wrote:
>
> I was poking around the Kylo repo and noticed that they appear to have a
> processor that is designed to directly launch a Spark job from NiFi (ie not
> just work w/ Livy). Any thoughts on reviewing and possibly bringing that
> into the core bundles since IIRC it's ASL 2.0?
>
> On Fri, May 11, 2018 at 10:27 AM dan young <da...@gmail.com> wrote:
>>
>> Awesome! NiFi is an amazing tool. As Joe mentioned the docs are a great
>> place to start along with the Hortonworks forums. Although we don't use HDP,
>> the content is still very relevant.
>>
>> Start simple and iterate...Install on your laptop and start moving bits!
>>
>> Regards
>>
>> Dano
>>
>>
>> On Fri, May 11, 2018, 7:54 AM Andrés Ivaldi <ia...@gmail.com> wrote:
>>>
>>> Hello, I'm new with Nifi,
>>>
>>> I'm working on a POC where flow must be created dinamically and result
>>> processed with spark, which would be the best way to start?
>>>
>>> I've never used Nifi before.
>>>
>>> Regards,
>>>
>>>
>>> --
>>> Ing. Ivaldi Andres
>
>
Re: First steps with Nifi
Posted by Damian Czaja <da...@nokia.com>.
Here is a great repo with a lot of information about NiFi and tools
around it created by the community:
https://github.com/jfrazee/awesome-nifi
There is also some processor to submit Spark Jobs:
https://github.com/qiansl127/nifi-spark-bundle
Best regards,
Damian
On 11.05.2018 16:48, Mike Thomsen wrote:
> I was poking around the Kylo repo and noticed that they appear to have
> a processor that is designed to directly launch a Spark job from NiFi
> (ie not just work w/ Livy). Any thoughts on reviewing and possibly
> bringing that into the core bundles since IIRC it's ASL 2.0?
>
> On Fri, May 11, 2018 at 10:27 AM dan young <danoyoung@gmail.com
> <ma...@gmail.com>> wrote:
>
> Awesome! NiFi is an amazing tool. As Joe mentioned the docs are a
> great place to start along with the Hortonworks forums. Although
> we don't use HDP, the content is still very relevant.
>
> Start simple and iterate...Install on your laptop and start moving
> bits!
>
> Regards
>
> Dano
>
>
> On Fri, May 11, 2018, 7:54 AM Andrés Ivaldi <iaivaldi@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hello, I'm new with Nifi,
>
> I'm working on a POC where flow must be created dinamically
> and result processed with spark, which would be the best way
> to start?
>
> I've never used Nifi before.
>
> Regards,
>
>
> --
> Ing. Ivaldi Andres
>
Re: First steps with Nifi
Posted by Mike Thomsen <mi...@gmail.com>.
I was poking around the Kylo repo and noticed that they appear to have a
processor that is designed to directly launch a Spark job from NiFi (ie not
just work w/ Livy). Any thoughts on reviewing and possibly bringing that
into the core bundles since IIRC it's ASL 2.0?
On Fri, May 11, 2018 at 10:27 AM dan young <da...@gmail.com> wrote:
> Awesome! NiFi is an amazing tool. As Joe mentioned the docs are a great
> place to start along with the Hortonworks forums. Although we don't use
> HDP, the content is still very relevant.
>
> Start simple and iterate...Install on your laptop and start moving bits!
>
> Regards
>
> Dano
>
>
> On Fri, May 11, 2018, 7:54 AM Andrés Ivaldi <ia...@gmail.com> wrote:
>
>> Hello, I'm new with Nifi,
>>
>> I'm working on a POC where flow must be created dinamically and result
>> processed with spark, which would be the best way to start?
>>
>> I've never used Nifi before.
>>
>> Regards,
>>
>>
>> --
>> Ing. Ivaldi Andres
>>
>
Re: First steps with Nifi
Posted by dan young <da...@gmail.com>.
Awesome! NiFi is an amazing tool. As Joe mentioned the docs are a great
place to start along with the Hortonworks forums. Although we don't use
HDP, the content is still very relevant.
Start simple and iterate...Install on your laptop and start moving bits!
Regards
Dano
On Fri, May 11, 2018, 7:54 AM Andrés Ivaldi <ia...@gmail.com> wrote:
> Hello, I'm new with Nifi,
>
> I'm working on a POC where flow must be created dinamically and result
> processed with spark, which would be the best way to start?
>
> I've never used Nifi before.
>
> Regards,
>
>
> --
> Ing. Ivaldi Andres
>
Re: First steps with Nifi
Posted by Andrés Ivaldi <ia...@gmail.com>.
Thanks Joe,
I'll start from that
On Fri, May 11, 2018 at 10:59 AM, Joe Witt <jo...@gmail.com> wrote:
> Andres
>
> Welcome to nifi. There are a lot of user guides, web pages, training
> materials, youtube videos, and such to help you get going. I'd
> definitely recommend starting from there to familiarize yourself with
> NiFi. Setup some kind of simple self contained local flow to try
> things out and get a feel for it yourself. Once you're comfortable
> there I'd then start adding in interaction with Spark.
>
> Thanks
>
> On Fri, May 11, 2018 at 9:53 AM, Andrés Ivaldi <ia...@gmail.com> wrote:
> > Hello, I'm new with Nifi,
> >
> > I'm working on a POC where flow must be created dinamically and result
> > processed with spark, which would be the best way to start?
> >
> > I've never used Nifi before.
> >
> > Regards,
> >
> > --
> > Ing. Ivaldi Andres
>
--
Ing. Ivaldi Andres
Re: First steps with Nifi
Posted by Joe Witt <jo...@gmail.com>.
Andres
Welcome to nifi. There are a lot of user guides, web pages, training
materials, youtube videos, and such to help you get going. I'd
definitely recommend starting from there to familiarize yourself with
NiFi. Setup some kind of simple self contained local flow to try
things out and get a feel for it yourself. Once you're comfortable
there I'd then start adding in interaction with Spark.
Thanks
On Fri, May 11, 2018 at 9:53 AM, Andrés Ivaldi <ia...@gmail.com> wrote:
> Hello, I'm new with Nifi,
>
> I'm working on a POC where flow must be created dinamically and result
> processed with spark, which would be the best way to start?
>
> I've never used Nifi before.
>
> Regards,
>
> --
> Ing. Ivaldi Andres