Nifi jobs support Mesos ?

[Timothy Findlay <tf...@prodevelop.com.au>]

Hi Folks,


Forgive me as I'm new to Nifi. The tool seems fantastic, the UI is super easy to use. It was easy to extend and build a custom processor which is great.


Looking at our processing, we're curious if we could use Nifi to replace some of our batch processes. Setting up some of the basic routines, scheduled by cron is all working.


However we would like the ability to scale jobs out, horizontally and distribute processing across a number of Mesos nodes.


Has anyone done this ? any notes, tips or documentation on how to set this up ?


Thanks.

Re: Nifi jobs support Mesos ?

[Oleg Zhurakousky <oz...@hortonworks.com>]

Timothy

In short it’s very possible. I personally haven’t done that, but I have written some custom code submitting jobs to YARN which I am assuming would be very similar. 
Would be happy and actually interested to take it a bit further. Could you elaborate on the use case a bit more? For example; Where is the source of data? How does Mesos know about where the data is? 
Sorry some of these question may be very dumb (not much experience with Mesos other then general understanding) and what responsibility do you envision NiFi to have over such process?

I’ll share my example with NiFi and Tez where a custom NiFi Processor would submit job to Tez and upon completion would create a FlowFile that would trigger another processor which would invoke another Tez job and so on. Tez essentially is a processing wrapper that uses YARN for resource management.
So I am assuming that something similar would be with Mesos. Am I on the right track?

Cheers
Oleg
 
> On Feb 1, 2016, at 4:53 PM, Timothy Findlay <tf...@prodevelop.com.au> wrote:
> 
> Hi Folks,
> 
> 
> Forgive me as I'm new to Nifi. The tool seems fantastic, the UI is super easy to use. It was easy to extend and build a custom processor which is great.
> 
> 
> Looking at our processing, we're curious if we could use Nifi to replace some of our batch processes. Setting up some of the basic routines, scheduled by cron is all working.
> 
> 
> However we would like the ability to scale jobs out, horizontally and distribute processing across a number of Mesos nodes.
> 
> 
> Has anyone done this ? any notes, tips or documentation on how to set this up ?
> 
> 
> Thanks.