Custom processors and restart

[Guido De Caso <gu...@medallia.com>]

Hi everyone,

I've been reading about Apache NiFi and trying it out at my company as a
data-pipeline-as-a-service initiative.

We are envisioning that third-parties, or at least different organizations
within my company should be able to write custom connectors.

As far as I've been reading around, including in previous posts on this
e-mail group, the current flow for doing so requires NiFi to be restarted.
In other platforms that we are considering (e.g., Spring Cloud Dataflow)
this can be achieved without restarting the platform itself.

NiFi has been so far our preferred choice to go forward, but the
must-restart-to-install-custom-processor requirement is making it a hard
sell internally.

Is "hot-load" something that the NiFi team sees in the roadmap? Or does
NiFi architecture make this fundamentally impossible to achieve?

Thanks in advance!

-- 
*Guido de Caso*
Software Architect  |  ❖ Medallia, Inc.

<http://medallia.com.ar/careers>
<https://www.linkedin.com/company/medallia-inc.>
<http://engineering.medallia.com/blog>  <https://twitter.com/MedalliaArg>
<https://www.facebook.com/medalliaargentina>

Re: Custom processors and restart

[Guido De Caso <gu...@medallia.com>]

Thanks Mark for answering so quickly!

Is this something that's being actively worked on? If so I can contribute
myself to the work streams


On Fri, May 4, 2018 at 5:16 PM, Mark Payne <ma...@hotmail.com> wrote:

> Guido,
>
> Dynamically deploying extensions is very much on the roadmap. There is
> actually a Feature Proposal [1]
> written that outlines some of the ideas about how it would work. The idea
> being that extensions would be
> published to an Extension Registry, and NiFi would then interact with that
> Extension Registry (or registries)
> to pull in the extensions at runtime as needed.
>
> Thanks
> -Mark
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/Extensi
> on+Repositories+%28aka+Extension+Registry%29+for+Dynamically
> -loaded+Extensions
> <https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+(aka+Extension+Registry)+for+Dynamically-loaded+Extensions>
>
>
> On May 4, 2018, at 4:01 PM, Guido De Caso <gu...@medallia.com> wrote:
>
> Hi everyone,
>
> I've been reading about Apache NiFi and trying it out at my company as a
> data-pipeline-as-a-service initiative.
>
> We are envisioning that third-parties, or at least different organizations
> within my company should be able to write custom connectors.
>
> As far as I've been reading around, including in previous posts on this
> e-mail group, the current flow for doing so requires NiFi to be restarted.
> In other platforms that we are considering (e.g., Spring Cloud Dataflow)
> this can be achieved without restarting the platform itself.
>
> NiFi has been so far our preferred choice to go forward, but the
> must-restart-to-install-custom-processor requirement is making it a hard
> sell internally.
>
> Is "hot-load" something that the NiFi team sees in the roadmap? Or does
> NiFi architecture make this fundamentally impossible to achieve?
>
> Thanks in advance!
>
> --
> *Guido de Caso*
> Software Architect  |  ❖ Medallia, Inc.
>
> <http://medallia.com.ar/careers>
> <https://www.linkedin.com/company/medallia-inc.>
> <http://engineering.medallia.com/blog>  <https://twitter.com/MedalliaArg>
> <https://www.facebook.com/medalliaargentina>
>
>
>


-- 
*Guido de Caso*
Software Architect  |  ❖ Medallia, Inc.

<http://medallia.com.ar/careers>
<https://www.linkedin.com/company/medallia-inc.>
<http://engineering.medallia.com/blog>  <https://twitter.com/MedalliaArg>
<https://www.facebook.com/medalliaargentina>

Re: Custom processors and restart

[Mark Payne <ma...@hotmail.com>]

Guido,

Dynamically deploying extensions is very much on the roadmap. There is actually a Feature Proposal [1]
written that outlines some of the ideas about how it would work. The idea being that extensions would be
published to an Extension Registry, and NiFi would then interact with that Extension Registry (or registries)
to pull in the extensions at runtime as needed.

Thanks
-Mark

[1] https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions<https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+(aka+Extension+Registry)+for+Dynamically-loaded+Extensions>


On May 4, 2018, at 4:01 PM, Guido De Caso <gu...@medallia.com>> wrote:

Hi everyone,

I've been reading about Apache NiFi and trying it out at my company as a data-pipeline-as-a-service initiative.

We are envisioning that third-parties, or at least different organizations within my company should be able to write custom connectors.

As far as I've been reading around, including in previous posts on this e-mail group, the current flow for doing so requires NiFi to be restarted. In other platforms that we are considering (e.g., Spring Cloud Dataflow) this can be achieved without restarting the platform itself.

NiFi has been so far our preferred choice to go forward, but the must-restart-to-install-custom-processor requirement is making it a hard sell internally.

Is "hot-load" something that the NiFi team sees in the roadmap? Or does NiFi architecture make this fundamentally impossible to achieve?

Thanks in advance!

--
Guido de Caso
Software Architect  |  ❖ Medallia, Inc.

[http://signature.medallia.com/google/med_logo_email.png]<http://medallia.com.ar/careers>  [http://signature.medallia.com/google/linkedin.png] <https://www.linkedin.com/company/medallia-inc.>  [http://signature.medallia.com/google/blog.png] <http://engineering.medallia.com/blog>  [http://signature.medallia.com/google/twitter.png] <https://twitter.com/MedalliaArg>  [http://signature.medallia.com/google/facebook.png] <https://www.facebook.com/medalliaargentina>