RE: [EXT] Airflow and NiFi

["Peter Wicks (pwicks)" <pw...@micron.com>]

User: "I want to change my production flow while it's running, you know, mid stream just route my content to a completely different flow, re-run data mid run through a new set of processors, fork it on the fly, you know, whatever I want anytime I want "

NiFi: "Yeah, I can do that"

Everyone Else: ".... I'm sorry... what?"

-----Original Message-----
From: Mike Thomsen <mi...@gmail.com> 
Sent: Thursday, August 15, 2019 2:00 PM
To: dev@nifi.apache.org
Subject: [EXT] Airflow and NiFi

Does anyone see any areas where the two can complement each other and where we might want to give users the ability to offload processing to Airflow?
Curious since our poking around the docs lead us to conclude it was probably more "Airflow vs Spark+Oozie" than really competing with NiFi.

Re: [EXT] Airflow and NiFi

[Muazma Zahid <mu...@gmail.com>]

Nifi is used for ETL and focus on transformations and data movement.
Airflow is good for orchestration. Both tools have scheduling in common.

Airflow is good at where the data is already available and you want to
control jobs.

If your focus is more like the data integertion, manipulation and
transformation and movement then NiFi is a great choice.


No doubt there is some overlap but ultimately it comes down to your use
case.


Thanks,

Muazma

On Thu, Aug 15, 2019, 1:15 PM Joe Witt <jo...@gmail.com> wrote:

> i blame spell check for ‘complimentary’
>
> On Thu, Aug 15, 2019 at 4:11 PM Joe Witt <jo...@gmail.com> wrote:
>
> > Love it.
> >
> > Airflow I see as an orchestrator of making other systems do things.  Nifi
> > i see as very focused on acquiring, manipulating, and routing data
> between
> > systems.
> >
> > In short, I believe theyre complimentary in a given architecture.
> Someone
> > always tries to use one to do both of the things but each is better at
> > their sweet spot.
> >
> > Thanks
> >
> > On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen <mi...@gmail.com>
> > wrote:
> >
> >> Yeah, pretty much.
> >>
> >> On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) <pwicks@micron.com
> >
> >> wrote:
> >>
> >> > User: "I want to change my production flow while it's running, you
> know,
> >> > mid stream just route my content to a completely different flow,
> re-run
> >> > data mid run through a new set of processors, fork it on the fly, you
> >> know,
> >> > whatever I want anytime I want "
> >> >
> >> > NiFi: "Yeah, I can do that"
> >> >
> >> > Everyone Else: ".... I'm sorry... what?"
> >> >
> >> > -----Original Message-----
> >> > From: Mike Thomsen <mi...@gmail.com>
> >> > Sent: Thursday, August 15, 2019 2:00 PM
> >> > To: dev@nifi.apache.org
> >> > Subject: [EXT] Airflow and NiFi
> >> >
> >> > Does anyone see any areas where the two can complement each other and
> >> > where we might want to give users the ability to offload processing to
> >> > Airflow?
> >> > Curious since our poking around the docs lead us to conclude it was
> >> > probably more "Airflow vs Spark+Oozie" than really competing with
> NiFi.
> >> >
> >>
> >
>

Re: [EXT] Airflow and NiFi

[Joe Witt <jo...@gmail.com>]

i blame spell check for ‘complimentary’

On Thu, Aug 15, 2019 at 4:11 PM Joe Witt <jo...@gmail.com> wrote:

> Love it.
>
> Airflow I see as an orchestrator of making other systems do things.  Nifi
> i see as very focused on acquiring, manipulating, and routing data between
> systems.
>
> In short, I believe theyre complimentary in a given architecture.  Someone
> always tries to use one to do both of the things but each is better at
> their sweet spot.
>
> Thanks
>
> On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> Yeah, pretty much.
>>
>> On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) <pw...@micron.com>
>> wrote:
>>
>> > User: "I want to change my production flow while it's running, you know,
>> > mid stream just route my content to a completely different flow, re-run
>> > data mid run through a new set of processors, fork it on the fly, you
>> know,
>> > whatever I want anytime I want "
>> >
>> > NiFi: "Yeah, I can do that"
>> >
>> > Everyone Else: ".... I'm sorry... what?"
>> >
>> > -----Original Message-----
>> > From: Mike Thomsen <mi...@gmail.com>
>> > Sent: Thursday, August 15, 2019 2:00 PM
>> > To: dev@nifi.apache.org
>> > Subject: [EXT] Airflow and NiFi
>> >
>> > Does anyone see any areas where the two can complement each other and
>> > where we might want to give users the ability to offload processing to
>> > Airflow?
>> > Curious since our poking around the docs lead us to conclude it was
>> > probably more "Airflow vs Spark+Oozie" than really competing with NiFi.
>> >
>>
>

Re: [EXT] Airflow and NiFi

[Joe Witt <jo...@gmail.com>]

Love it.

Airflow I see as an orchestrator of making other systems do things.  Nifi i
see as very focused on acquiring, manipulating, and routing data between
systems.

In short, I believe theyre complimentary in a given architecture.  Someone
always tries to use one to do both of the things but each is better at
their sweet spot.

Thanks

On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen <mi...@gmail.com> wrote:

> Yeah, pretty much.
>
> On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> > User: "I want to change my production flow while it's running, you know,
> > mid stream just route my content to a completely different flow, re-run
> > data mid run through a new set of processors, fork it on the fly, you
> know,
> > whatever I want anytime I want "
> >
> > NiFi: "Yeah, I can do that"
> >
> > Everyone Else: ".... I'm sorry... what?"
> >
> > -----Original Message-----
> > From: Mike Thomsen <mi...@gmail.com>
> > Sent: Thursday, August 15, 2019 2:00 PM
> > To: dev@nifi.apache.org
> > Subject: [EXT] Airflow and NiFi
> >
> > Does anyone see any areas where the two can complement each other and
> > where we might want to give users the ability to offload processing to
> > Airflow?
> > Curious since our poking around the docs lead us to conclude it was
> > probably more "Airflow vs Spark+Oozie" than really competing with NiFi.
> >
>

Re: [EXT] Airflow and NiFi

[Mike Thomsen <mi...@gmail.com>]

Yeah, pretty much.

On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) <pw...@micron.com>
wrote:

> User: "I want to change my production flow while it's running, you know,
> mid stream just route my content to a completely different flow, re-run
> data mid run through a new set of processors, fork it on the fly, you know,
> whatever I want anytime I want "
>
> NiFi: "Yeah, I can do that"
>
> Everyone Else: ".... I'm sorry... what?"
>
> -----Original Message-----
> From: Mike Thomsen <mi...@gmail.com>
> Sent: Thursday, August 15, 2019 2:00 PM
> To: dev@nifi.apache.org
> Subject: [EXT] Airflow and NiFi
>
> Does anyone see any areas where the two can complement each other and
> where we might want to give users the ability to offload processing to
> Airflow?
> Curious since our poking around the docs lead us to conclude it was
> probably more "Airflow vs Spark+Oozie" than really competing with NiFi.
>