Questions about NiFi / flow based programming

[Joe Witt <jo...@gmail.com>]

Hello

These questions were asked in forum on flow based programming so I thought
it best to redirect it here to the apache nifi (incubating) mailing list.

Question: How does it fare with GC Gitter and use of off heap memory to
prevent GC Gitter?

Response: Considerable effort has gone into and will continue to go into
ensuring we are as complimentary to the Java GC processes as possible.
We've done a great deal of design consideration to make this work well and
have considerable operational experience to help us understand if we're on
the right track in this regard.  That said, it is a framework and folks can
build their own processes.  As such for processors built which do not
consider/honor Java memory pressures it can create challenges.  We do also
some interesting things.  While we generally try to keep in-memory the
pointers and the metadata/attributes about the objects we're proccessing
(not their content) we do at times swap them out to some underlying
off-heap storage.  This greatly reduces Java memory pressure even in
conditions of massive backlogs which can at times occur.



Question: Will you have a web editor which can be used in web based
projects?

Response: NiFi itself runs as a full application - it is not meant to be
embedded into another application.  NiFi is controlled and observed through
a web-client we provide but any client that can talk to our HTTP-based API
will be sufficient.



Question: Will you able to use NoFlo UI with this as an alternative editor?
Will you support this in the future?

Response: I simply do not know enough about this to give you a good answer
but if it can talk to our HTTP API then possibly.



Question: Will the editor be a projectional editor / structure editor?

Response: Those were not terms I was familiar with.  But some googling has
me thinking it is more of a projectional editor.  I am prepared to learn I
am wrong here though ;-)



Question: Is it possible to extend this to more general Dataflow
programming using both Visual and text based language? Like: Microsoft
Dryad, Netflix Mantis, Apache Flink, Verilog, VHDL, CAL Actor Language, etc.

Response:  As of yet it is unclear if there is a good fit.  One important
difference between NiFi and many of the other systems I am aware of is the
notion of runtime/live modification versus design and deploy.  For many
other systems there is a language in which you design a flow and
compile/deploy.  With NiFi it is more like molding clay.  As you press in,
the clay changes shape.  This difference is important as it lends itself to
a vastly different set of possible user experiences.



Question: Will you support data connection and feed handlers for important
domains out of the box? E.g FIX Protocol handlers for trading, OpenMAMA
support, etc.

Response: Support for connectors will presumably occur at the will of the
community and as licensing permits.  We would love to see the range of
connectors provided grow based on contributions and feedback.



Question: Any plans to use a language work bench (
https://www.jetbrains.com/mps/, http://mbeddr.com/,
https://github.com/JetBrains/Nitra) so this can be ported outside the JVM
while maintaining the same code base?

Response: There are no plans for support outside the JVM at this time.



Question: Will this support data flow spreadsheet paradigm? E.g.
http://www.ankhor.com/en/

Response: I am not sure it applies but as we do a better job of
communicating, describing, and making NiFi available it will be interesting
to learn more.



Question: Will this support real time / streaming DataFrames? E.g.
http://ddf.io/, borrowing concepts from Pandas, R DataFrame,
https://www.quantrix.com/en/

Response: It is unclear if DataFrames specifically makes sense for us to
support.  It appears they have this already well supported on ideal
platforms like Apache Spark and others.  We do not replace or compete with
complex event processing engines - we compliment them through
orchestration/data flow management into and out of these environments and
the wide range of simple event processing requirements which are best done
in more simplified models.

Thanks
Joe