Apache NIFI vs Spring XD

[pmunis <pa...@newwave-technologies.com>]

Trying to figure out how apache nifi is different from spring XD (flows)?  



--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Apache-NIFI-vs-Spring-XD-tp1963.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.

Re: Apache NIFI vs Spring XD

[Tony Kurc <tr...@gmail.com>]

Are you asking if the apache nifi (incubating) development community is
attempting to determine the differences? Or are you asking if we will
enumerate the differences for you?

On Thu, Jun 18, 2015 at 3:01 PM, pmunis <
patrick.munis@newwave-technologies.com> wrote:

> Trying to figure out how apache nifi is different from spring XD (flows)?
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Apache-NIFI-vs-Spring-XD-tp1963.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>

Re: Apache NIFI vs Spring XD

[Joe Witt <jo...@gmail.com>]

Patrick,

I think Tony did a nice job of describing the potentially different
focus areas/motivations for each.  Ultimately they both do appear to
live in similar or overlapping parts of the architecture.  So it comes
down to feature differences, ideal use cases/strengths, and design
tradeoffs against what you’re trying to do.  I’m not a Spring XD
expert so rather than give some biased comparison I will simply focus
on a few key NiFi strengths that are important to understand.  If
these strengths play to what you’re trying to do I think NiFi will be
quite competitive as an option against any solution.  How these
compare to Spring XD is best left to Spring XD knowledgable folks.


While NiFi can interface with messaging systems like Kafka, ActiveMQ,
etc.., NiFi doesn’t depend an external messaging system such as
ActiveMQ or Kafka for its management of data in the flow.  Such an
approach would limit NiFi’s ability to deal with potentially very
large (many GB+) messages/objects/events.  NiFi also can prioritize
data in queues using configurable prioritization schemes.  This is not
something you will get in an insertion order messaging system for
example.  NiFi’s data repository is local to a given node and thus all
data interactions with those repositories are not remote procedure
calls.  If you’re just doing pure messaging/transport then this isn’t
exciting.  But if you’re having to do some transformations,
conversions, enrichments, compression, encryption, routing on content,
etc.. then you’re going to be really happy with how NiFi attacks this
problem.  NiFi’s repositories enable both copy-on-write and
pass-by-reference which are extremely important for being IO
efficient.


NiFi has data lineage baked into the core.  This is extremely powerful
and drives numerous compelling user experience scenarios including
troubleshooting workflows by stepping through time much like you would
in a debugger, fine-grained replay and content viewing, and audit
trails which are valuable for a number of use cases.


NiFi enables fine-grained interactive command and control through our
REST API (not just at a coarse grained job/stream level) whereby you
can alter specific points of the flow without having to stop unrelated
pieces.  NiFi provides a powerful User Interface and experience around
that REST API to visualize, understand, and alter the flow.  The
notion of observing state/status/history is nicely integrated into
this view.  We want to encourage iterative changes and direct
cause/effect.


There are certainly many more areas to highlight.  But this is
probably enough to get started if you want to have a discussion.  If
there are specific strengths of Spring XD you’re curious about
comparing to NiFi please advise what those are and the use cases for
which they’re compelling and we can have a more focused discussion of
differences.


Thanks

Joe

On Thu, Jun 18, 2015 at 7:38 PM, Tony Kurc <tr...@gmail.com> wrote:
> I suppose the easiest way to explain differences is scope. I can most
> easily talk about Apache NiFi scope, which is, as I would put it, a tool
> and platform for getting data from here to there (our incubator proposal
> explains scope well [1]), both at the edge (ingestion) and at the core
> (transport). There are tools built in to track data and support more cases
> at the edge by supplying one's own services and processors. It also has a
> very operator-centric UI for building and maintaining flows.
>
> The scope for Spark XD appears to be more of an application
> orchestration/integration framework (analytic glue). From my perspective,
> the UIs and programming interfaces are very programmer centric. It also has
> some overlap with Apache Falcon [2].
>
> There is overlap with Spark XD and NiFi. Both do ingest (into a myriad of
> sinks). Both can do 'analysis', but I don't feel like the scope of NiFi is
> to be a general purpose analysis platform.  But there certainly are lots of
> interesting things you can do with data while it goes from here.
>
>
> [1] http://wiki.apache.org/incubator/NiFiProposal
> [2] http://falcon.apache.org/
>
> On Thu, Jun 18, 2015 at 4:18 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> I will be happy to send along some diffs.  Will put that together soon.
>> On Jun 18, 2015 6:12 PM, <es...@gmail.com> wrote:
>>
>> > I'd be interested in seeing a list of differences too!
>> >
>> > A
>> >
>> > Sent from my iPhone
>> >
>> > > On Jun 18, 2015, at 6:01 PM, pmunis <
>> > patrick.munis@newwave-technologies.com> wrote:
>> > >
>> > > Trying to figure out how apache nifi is different from spring XD
>> (flows)?
>> > >
>> > >
>> > >
>> > > --
>> > > View this message in context:
>> >
>> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Apache-NIFI-vs-Spring-XD-tp1963.html
>> > > Sent from the Apache NiFi (incubating) Developer List mailing list
>> > archive at Nabble.com.
>> >
>>

Re: Apache NIFI vs Spring XD

[Tony Kurc <tr...@gmail.com>]

I suppose the easiest way to explain differences is scope. I can most
easily talk about Apache NiFi scope, which is, as I would put it, a tool
and platform for getting data from here to there (our incubator proposal
explains scope well [1]), both at the edge (ingestion) and at the core
(transport). There are tools built in to track data and support more cases
at the edge by supplying one's own services and processors. It also has a
very operator-centric UI for building and maintaining flows.

The scope for Spark XD appears to be more of an application
orchestration/integration framework (analytic glue). From my perspective,
the UIs and programming interfaces are very programmer centric. It also has
some overlap with Apache Falcon [2].

There is overlap with Spark XD and NiFi. Both do ingest (into a myriad of
sinks). Both can do 'analysis', but I don't feel like the scope of NiFi is
to be a general purpose analysis platform.  But there certainly are lots of
interesting things you can do with data while it goes from here.


[1] http://wiki.apache.org/incubator/NiFiProposal
[2] http://falcon.apache.org/

On Thu, Jun 18, 2015 at 4:18 PM, Joe Witt <jo...@gmail.com> wrote:

> I will be happy to send along some diffs.  Will put that together soon.
> On Jun 18, 2015 6:12 PM, <es...@gmail.com> wrote:
>
> > I'd be interested in seeing a list of differences too!
> >
> > A
> >
> > Sent from my iPhone
> >
> > > On Jun 18, 2015, at 6:01 PM, pmunis <
> > patrick.munis@newwave-technologies.com> wrote:
> > >
> > > Trying to figure out how apache nifi is different from spring XD
> (flows)?
> > >
> > >
> > >
> > > --
> > > View this message in context:
> >
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Apache-NIFI-vs-Spring-XD-tp1963.html
> > > Sent from the Apache NiFi (incubating) Developer List mailing list
> > archive at Nabble.com.
> >
>

Re: Apache NIFI vs Spring XD

[Joe Witt <jo...@gmail.com>]

I will be happy to send along some diffs.  Will put that together soon.
On Jun 18, 2015 6:12 PM, <es...@gmail.com> wrote:

> I'd be interested in seeing a list of differences too!
>
> A
>
> Sent from my iPhone
>
> > On Jun 18, 2015, at 6:01 PM, pmunis <
> patrick.munis@newwave-technologies.com> wrote:
> >
> > Trying to figure out how apache nifi is different from spring XD (flows)?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Apache-NIFI-vs-Spring-XD-tp1963.html
> > Sent from the Apache NiFi (incubating) Developer List mailing list
> archive at Nabble.com.
>

Re: Apache NIFI vs Spring XD

[es...@gmail.com]

I'd be interested in seeing a list of differences too!

A

Sent from my iPhone

> On Jun 18, 2015, at 6:01 PM, pmunis <pa...@newwave-technologies.com> wrote:
> 
> Trying to figure out how apache nifi is different from spring XD (flows)?  
> 
> 
> 
> --
> View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Apache-NIFI-vs-Spring-XD-tp1963.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.