Proper use of parameter contexts?

[James McMahon <js...@gmail.com>]

Parameter contexts appear to be a powerful feature with many uses. For
example, anaging environment-dependent global variables that must change as
flows are promoted from development, to integration, to production. Also,
preserving or managing parameters that are sensitive - such as settings for
passwords. Those both seem to be reasonable uses for parameters that are,
well, context dependent.

From the few opportunities I've had to use them so far they also seem to be
for parameters that transcend Process Group boundaries. You want to use
parameter contexts for things that are global. You should avoid creating
parameter contexts for each and every Process Group.

I mention this because I notice a recent tendency on my current team to
park flow-specific attribute settings in parameter context maps associated
with each Process Group flow. I'm not quite yet experienced enough with
parameter contexts to offer any argument against this other than it may
tend to lead to many contexts that each become very large, maybe even
redundant over time.

Are there drawbacks to employing parameter contexts for each and every
individual Process Group? Performance-related, configuration management,
other...?

Anyone work through this on their teams and determine reasonable best
practices for using parameter contexts vs. Nifi variables vs. flow
attributes?

Thank you for any insights.

Re: Proper use of parameter contexts?

[Joe Gresock <jg...@gmail.com>]

[1] also has a discussion on using parameter context inheritance to
overcome some of the problems you're describing.

[1]
https://bryanbende.com/development/2021/11/08/apache-nifi-1-15-0-parameter-context-inheritance

On Fri, Jun 3, 2022 at 11:25 AM James McMahon <js...@gmail.com> wrote:

> I see. Sounds like I should accept parameter contexts for each process
> group. Thank you for replying Bryan.
>
> On Fri, Jun 3, 2022 at 11:10 AM Bryan Bende <bb...@gmail.com> wrote:
>
>> Parameter contexts are basically a replacement for variables because
>> of several limitations of variables. Mainly variables don't support
>> sensitive values and they are also coupled to expression language
>> which makes it unclear what the expression is actually going to
>> resolve to (variable, FF attribute, system prop, env var, etc).
>>
>> Parameter contexts are definitely flow related, they are the
>> environmental configuration to make a flow run. So if you have a
>> single multi-tenant NiFi cluster where there are many top-level PGs
>> used by different teams that represent your "flows" then you'd
>> probably have a parameter context for each of these.
>>
>> There is an opportunity for sharing common parameters across contexts
>> through the concept of parameter context inheritance, so there could
>> be a base context that all child contexts inherit some global
>> parameters from.
>>
>>
>> On Fri, Jun 3, 2022 at 10:54 AM James McMahon <js...@gmail.com>
>> wrote:
>> >
>> > Parameter contexts appear to be a powerful feature with many uses. For
>> example, anaging environment-dependent global variables that must change as
>> flows are promoted from development, to integration, to production. Also,
>> preserving or managing parameters that are sensitive - such as settings for
>> passwords. Those both seem to be reasonable uses for parameters that are,
>> well, context dependent.
>> >
>> > From the few opportunities I've had to use them so far they also seem
>> to be for parameters that transcend Process Group boundaries. You want to
>> use parameter contexts for things that are global. You should avoid
>> creating parameter contexts for each and every Process Group.
>> >
>> > I mention this because I notice a recent tendency on my current team to
>> park flow-specific attribute settings in parameter context maps associated
>> with each Process Group flow. I'm not quite yet experienced enough with
>> parameter contexts to offer any argument against this other than it may
>> tend to lead to many contexts that each become very large, maybe even
>> redundant over time.
>> >
>> > Are there drawbacks to employing parameter contexts for each and every
>> individual Process Group? Performance-related, configuration management,
>> other...?
>> >
>> > Anyone work through this on their teams and determine reasonable best
>> practices for using parameter contexts vs. Nifi variables vs. flow
>> attributes?
>> >
>> > Thank you for any insights.
>>
>

Re: Proper use of parameter contexts?

[James McMahon <js...@gmail.com>]

I see. Sounds like I should accept parameter contexts for each process
group. Thank you for replying Bryan.

On Fri, Jun 3, 2022 at 11:10 AM Bryan Bende <bb...@gmail.com> wrote:

> Parameter contexts are basically a replacement for variables because
> of several limitations of variables. Mainly variables don't support
> sensitive values and they are also coupled to expression language
> which makes it unclear what the expression is actually going to
> resolve to (variable, FF attribute, system prop, env var, etc).
>
> Parameter contexts are definitely flow related, they are the
> environmental configuration to make a flow run. So if you have a
> single multi-tenant NiFi cluster where there are many top-level PGs
> used by different teams that represent your "flows" then you'd
> probably have a parameter context for each of these.
>
> There is an opportunity for sharing common parameters across contexts
> through the concept of parameter context inheritance, so there could
> be a base context that all child contexts inherit some global
> parameters from.
>
>
> On Fri, Jun 3, 2022 at 10:54 AM James McMahon <js...@gmail.com>
> wrote:
> >
> > Parameter contexts appear to be a powerful feature with many uses. For
> example, anaging environment-dependent global variables that must change as
> flows are promoted from development, to integration, to production. Also,
> preserving or managing parameters that are sensitive - such as settings for
> passwords. Those both seem to be reasonable uses for parameters that are,
> well, context dependent.
> >
> > From the few opportunities I've had to use them so far they also seem to
> be for parameters that transcend Process Group boundaries. You want to use
> parameter contexts for things that are global. You should avoid creating
> parameter contexts for each and every Process Group.
> >
> > I mention this because I notice a recent tendency on my current team to
> park flow-specific attribute settings in parameter context maps associated
> with each Process Group flow. I'm not quite yet experienced enough with
> parameter contexts to offer any argument against this other than it may
> tend to lead to many contexts that each become very large, maybe even
> redundant over time.
> >
> > Are there drawbacks to employing parameter contexts for each and every
> individual Process Group? Performance-related, configuration management,
> other...?
> >
> > Anyone work through this on their teams and determine reasonable best
> practices for using parameter contexts vs. Nifi variables vs. flow
> attributes?
> >
> > Thank you for any insights.
>

Re: Proper use of parameter contexts?

[Bryan Bende <bb...@gmail.com>]

Parameter contexts are basically a replacement for variables because
of several limitations of variables. Mainly variables don't support
sensitive values and they are also coupled to expression language
which makes it unclear what the expression is actually going to
resolve to (variable, FF attribute, system prop, env var, etc).

Parameter contexts are definitely flow related, they are the
environmental configuration to make a flow run. So if you have a
single multi-tenant NiFi cluster where there are many top-level PGs
used by different teams that represent your "flows" then you'd
probably have a parameter context for each of these.

There is an opportunity for sharing common parameters across contexts
through the concept of parameter context inheritance, so there could
be a base context that all child contexts inherit some global
parameters from.


On Fri, Jun 3, 2022 at 10:54 AM James McMahon <js...@gmail.com> wrote:
>
> Parameter contexts appear to be a powerful feature with many uses. For example, anaging environment-dependent global variables that must change as flows are promoted from development, to integration, to production. Also, preserving or managing parameters that are sensitive - such as settings for passwords. Those both seem to be reasonable uses for parameters that are, well, context dependent.
>
> From the few opportunities I've had to use them so far they also seem to be for parameters that transcend Process Group boundaries. You want to use parameter contexts for things that are global. You should avoid creating parameter contexts for each and every Process Group.
>
> I mention this because I notice a recent tendency on my current team to park flow-specific attribute settings in parameter context maps associated with each Process Group flow. I'm not quite yet experienced enough with parameter contexts to offer any argument against this other than it may tend to lead to many contexts that each become very large, maybe even redundant over time.
>
> Are there drawbacks to employing parameter contexts for each and every individual Process Group? Performance-related, configuration management, other...?
>
> Anyone work through this on their teams and determine reasonable best practices for using parameter contexts vs. Nifi variables vs. flow attributes?
>
> Thank you for any insights.