Nifi driven by external configuration

["Carlos Manuel Fernandes (DSI)" <ca...@cgd.pt>]

Nifi simple principles of flow, attributes of flow ,  processes and  properties of  processors are very powerful and permit a lot of great combinations.

The problem i see,  is the tendency to repeat flows to make the  same thing, because some properties are hardwire (properties without expression language)  and doesn't exist a processor to read External configuration to map to attributes , which can be properties  in processors.

In my Use Cases  I try to address this problem making services with this skeleton :
HandleHttpRequest (receive some Key)  -> GetExternalConfig (Using  Key) -> Others Nifi Procs with  properties based on attributes read from external config -> HandleHttpResponse
GetExternalConfig is a custom script processor, which transform a sql Query (all the columns of first row)  in attributes of the current Flow.

In this arrangement , in the zone of "Other Nifi Procs" ,  if properties of processors don't permit expression language is not possible to use the configuration already read, which is annoying.
Till now, I had problems with SplitText (Line Split Count) and CSVRecordSetWriter (value separator, time format, etc), but the problem is general.

If this make sense for more people in the community , we can think to:

1-      Create a Nifi Processor to read external Configuration from relational Databases  to Attributes. (probably make sense, read from another non relational sources)

2-      By default all the processors properties must admit expression language (probably some technical ones don't make sense)

If not, I continue  with my custom GetExternalConfig and blaming when I find a property without expression language :)

Thanks

Carlos

Re: Nifi driven by external configuration

[Charlie Meyer <ch...@civitaslearning.com>]

To solve this problem, I have written code that sets configuration
properties in the variable registry on each nifi instance, then each flow
that is provisioned has its configuration reference those variables. This
way we are able to deploy nifi instances for different environments, etc
but maintain common logic.

On Wed, Apr 18, 2018 at 6:39 AM, Carlos Manuel Fernandes (DSI) <
carlos.antonio.fernandes@cgd.pt> wrote:

> Nifi simple principles of flow, attributes of flow ,  processes and
>  properties of  processors are very powerful and permit a lot of great
> combinations.
>
>
>
> The problem i see,  is the tendency to repeat flows to make the  same
> thing, because some properties are hardwire (properties without expression
> language)  and doesn’t exist a processor to read External configuration to
> map to attributes , which can be properties  in processors.
>
>
>
> In my Use Cases  I try to address this problem making services with this
> skeleton :
>
> HandleHttpRequest (receive some Key)  -> GetExternalConfig (Using  Key) ->
> Others Nifi Procs with  properties based on attributes read from external
> config -> HandleHttpResponse
>
> GetExternalConfig is a custom script processor, which transform a sql
> Query (all the columns of first row)  in attributes of the current Flow.
>
>
>
> In this arrangement , in the zone of “Other Nifi Procs” ,  if properties
> of processors don’t permit expression language is not possible to use the
> configuration already read, which is annoying.
>
> Till now, I had problems with SplitText (Line Split Count*) *and
> CSVRecordSetWriter (value separator, time format, etc), but the problem is
> general.
>
>
>
> If this make sense for more people in the community , we can think to:
>
> 1-      Create a Nifi Processor to read external Configuration from
> relational Databases  to Attributes. (probably make sense, read from
> another non relational sources)
>
> 2-      By default all the processors properties must admit expression
> language (probably some technical ones don’t make sense)
>
>
>
> If not, I continue  with my custom GetExternalConfig and blaming when I
> find a property without expression language J
>
>
>
> Thanks
>
>
>
> Carlos
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>