You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2022/02/04 12:38:12 UTC
Parameters and Process Groups
We have just started to employ parameter contexts rather than environment
variables. I have read that you can only have one parameter context for a
process group. We often have a process group that employs other process
groups in its flow, let's call the parent group A and the called group B.
For my questions, there is a parameter context associated with the parent
process group A.
Is there a concept of parameter context inheritance? Is there parameter
context override? More specifically:
In the case where the internal called process group B has no parameter
context set, will it inherit the parameter context from the parent process
group A?
In the case where the internal called process group B has a parameter
context set that shares no parameter names in common with the context of A,
is B aware of a context that is A context + B context?
In the case where the internal called process group B has a parameter
context set that shares one or more parameter names in common with the
context of A, is B aware of a context that is A context+B context, but its
common parameter overrides that from A?
Re: Parameters and Process Groups
Posted by James McMahon <js...@gmail.com>.
We are not yet nifi 1.15.x, we are 1.14.x. So this inheritance is not yet
quite possible for us. Until we make that version upgrade, would it work if
I use a parameter context at my NiFi Flow canvas level for all my common
parameters that I will want to change based on environment (int, prod,
etc), and employ environment variables at the Process Group level to set
variables unique to the group?
Can parameter contexts and environment variables both be employed? Is there
any reason I can't reference parameters and environment variables in the
same flow?
Jim
On Fri, Feb 4, 2022 at 8:04 AM James McMahon <js...@gmail.com> wrote:
> This is extremely helpful, Joe. I'm going to read through the link Pierre
> provided to learn more and I'll do as you recommend.
> Thank you both very much, Pierre and Joe.
> Jim
>
> On Fri, Feb 4, 2022 at 7:52 AM Joe Gresock <jg...@gmail.com> wrote:
>
>> Hi James,
>>
>> I'd like to add that while there is a concept of parameter context
>> inheritance in NiFi 1.15.x, as Pierre linked in the above article, this
>> only applies to a single Process Group. There is not currently a concept
>> of inheritance of parameters through the nesting of Process Groups.
>> Therefore, you have to explicitly set a Parameter Context in each Process
>> Group, no matter which level, in order for it to "see" any Parameters. In
>> your example, if process group B has no parameter context, it has no
>> parameters, regardless of how process group A is configured.
>>
>> If you'd like to use the concept of parameter context inheritance, what
>> you can do is the following:
>> - Create Parameter Context A for process group A
>> - Create Parameter Context B with any parameters you want in process
>> group B, including any that might want to override parameters in A. In
>> this context, add Context A to the Inherited Parameter Contexts.
>>
>> Assign Context A to group A, and Context B to group B. This way, Context
>> A will get all the parameters from Context A, while B will get the ones
>> from Context A + the ones directly in Context B. Context B's parameters
>> will override any inherited from Context A, since direct parameters always
>> override inherited ones (as described in the blog post).
>>
>> Cheers,
>> Joe
>>
>> On Fri, Feb 4, 2022 at 7:40 AM Pierre Villard <
>> pierre.villard.fr@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> You probably want to read this:
>>>
>>> https://bryanbende.com/development/2021/11/08/apache-nifi-1-15-0-parameter-context-inheritance
>>>
>>> Thanks,
>>> Pierre
>>>
>>> Le ven. 4 févr. 2022 à 13:38, James McMahon <js...@gmail.com> a
>>> écrit :
>>>
>>>> We have just started to employ parameter contexts rather than
>>>> environment variables. I have read that you can only have one parameter
>>>> context for a process group. We often have a process group that employs
>>>> other process groups in its flow, let's call the parent group A and the
>>>> called group B. For my questions, there is a parameter context associated
>>>> with the parent process group A.
>>>>
>>>> Is there a concept of parameter context inheritance? Is there parameter
>>>> context override? More specifically:
>>>>
>>>> In the case where the internal called process group B has no parameter
>>>> context set, will it inherit the parameter context from the parent process
>>>> group A?
>>>>
>>>> In the case where the internal called process group B has a parameter
>>>> context set that shares no parameter names in common with the context of A,
>>>> is B aware of a context that is A context + B context?
>>>>
>>>> In the case where the internal called process group B has a parameter
>>>> context set that shares one or more parameter names in common with the
>>>> context of A, is B aware of a context that is A context+B context, but its
>>>> common parameter overrides that from A?
>>>>
>>>
Re: Parameters and Process Groups
Posted by James McMahon <js...@gmail.com>.
This is extremely helpful, Joe. I'm going to read through the link Pierre
provided to learn more and I'll do as you recommend.
Thank you both very much, Pierre and Joe.
Jim
On Fri, Feb 4, 2022 at 7:52 AM Joe Gresock <jg...@gmail.com> wrote:
> Hi James,
>
> I'd like to add that while there is a concept of parameter context
> inheritance in NiFi 1.15.x, as Pierre linked in the above article, this
> only applies to a single Process Group. There is not currently a concept
> of inheritance of parameters through the nesting of Process Groups.
> Therefore, you have to explicitly set a Parameter Context in each Process
> Group, no matter which level, in order for it to "see" any Parameters. In
> your example, if process group B has no parameter context, it has no
> parameters, regardless of how process group A is configured.
>
> If you'd like to use the concept of parameter context inheritance, what
> you can do is the following:
> - Create Parameter Context A for process group A
> - Create Parameter Context B with any parameters you want in process group
> B, including any that might want to override parameters in A. In this
> context, add Context A to the Inherited Parameter Contexts.
>
> Assign Context A to group A, and Context B to group B. This way, Context
> A will get all the parameters from Context A, while B will get the ones
> from Context A + the ones directly in Context B. Context B's parameters
> will override any inherited from Context A, since direct parameters always
> override inherited ones (as described in the blog post).
>
> Cheers,
> Joe
>
> On Fri, Feb 4, 2022 at 7:40 AM Pierre Villard <pi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> You probably want to read this:
>>
>> https://bryanbende.com/development/2021/11/08/apache-nifi-1-15-0-parameter-context-inheritance
>>
>> Thanks,
>> Pierre
>>
>> Le ven. 4 févr. 2022 à 13:38, James McMahon <js...@gmail.com> a
>> écrit :
>>
>>> We have just started to employ parameter contexts rather than
>>> environment variables. I have read that you can only have one parameter
>>> context for a process group. We often have a process group that employs
>>> other process groups in its flow, let's call the parent group A and the
>>> called group B. For my questions, there is a parameter context associated
>>> with the parent process group A.
>>>
>>> Is there a concept of parameter context inheritance? Is there parameter
>>> context override? More specifically:
>>>
>>> In the case where the internal called process group B has no parameter
>>> context set, will it inherit the parameter context from the parent process
>>> group A?
>>>
>>> In the case where the internal called process group B has a parameter
>>> context set that shares no parameter names in common with the context of A,
>>> is B aware of a context that is A context + B context?
>>>
>>> In the case where the internal called process group B has a parameter
>>> context set that shares one or more parameter names in common with the
>>> context of A, is B aware of a context that is A context+B context, but its
>>> common parameter overrides that from A?
>>>
>>
Re: Parameters and Process Groups
Posted by Joe Gresock <jg...@gmail.com>.
Hi James,
I'd like to add that while there is a concept of parameter context
inheritance in NiFi 1.15.x, as Pierre linked in the above article, this
only applies to a single Process Group. There is not currently a concept
of inheritance of parameters through the nesting of Process Groups.
Therefore, you have to explicitly set a Parameter Context in each Process
Group, no matter which level, in order for it to "see" any Parameters. In
your example, if process group B has no parameter context, it has no
parameters, regardless of how process group A is configured.
If you'd like to use the concept of parameter context inheritance, what you
can do is the following:
- Create Parameter Context A for process group A
- Create Parameter Context B with any parameters you want in process group
B, including any that might want to override parameters in A. In this
context, add Context A to the Inherited Parameter Contexts.
Assign Context A to group A, and Context B to group B. This way, Context A
will get all the parameters from Context A, while B will get the ones from
Context A + the ones directly in Context B. Context B's parameters will
override any inherited from Context A, since direct parameters always
override inherited ones (as described in the blog post).
Cheers,
Joe
On Fri, Feb 4, 2022 at 7:40 AM Pierre Villard <pi...@gmail.com>
wrote:
> Hi,
>
> You probably want to read this:
>
> https://bryanbende.com/development/2021/11/08/apache-nifi-1-15-0-parameter-context-inheritance
>
> Thanks,
> Pierre
>
> Le ven. 4 févr. 2022 à 13:38, James McMahon <js...@gmail.com> a
> écrit :
>
>> We have just started to employ parameter contexts rather than environment
>> variables. I have read that you can only have one parameter context for a
>> process group. We often have a process group that employs other process
>> groups in its flow, let's call the parent group A and the called group B.
>> For my questions, there is a parameter context associated with the parent
>> process group A.
>>
>> Is there a concept of parameter context inheritance? Is there parameter
>> context override? More specifically:
>>
>> In the case where the internal called process group B has no parameter
>> context set, will it inherit the parameter context from the parent process
>> group A?
>>
>> In the case where the internal called process group B has a parameter
>> context set that shares no parameter names in common with the context of A,
>> is B aware of a context that is A context + B context?
>>
>> In the case where the internal called process group B has a parameter
>> context set that shares one or more parameter names in common with the
>> context of A, is B aware of a context that is A context+B context, but its
>> common parameter overrides that from A?
>>
>
Re: Parameters and Process Groups
Posted by James McMahon <js...@gmail.com>.
Looks promising, thank you Pierre.
Jim
On Fri, Feb 4, 2022 at 7:40 AM Pierre Villard <pi...@gmail.com>
wrote:
> Hi,
>
> You probably want to read this:
>
> https://bryanbende.com/development/2021/11/08/apache-nifi-1-15-0-parameter-context-inheritance
>
> Thanks,
> Pierre
>
> Le ven. 4 févr. 2022 à 13:38, James McMahon <js...@gmail.com> a
> écrit :
>
>> We have just started to employ parameter contexts rather than environment
>> variables. I have read that you can only have one parameter context for a
>> process group. We often have a process group that employs other process
>> groups in its flow, let's call the parent group A and the called group B.
>> For my questions, there is a parameter context associated with the parent
>> process group A.
>>
>> Is there a concept of parameter context inheritance? Is there parameter
>> context override? More specifically:
>>
>> In the case where the internal called process group B has no parameter
>> context set, will it inherit the parameter context from the parent process
>> group A?
>>
>> In the case where the internal called process group B has a parameter
>> context set that shares no parameter names in common with the context of A,
>> is B aware of a context that is A context + B context?
>>
>> In the case where the internal called process group B has a parameter
>> context set that shares one or more parameter names in common with the
>> context of A, is B aware of a context that is A context+B context, but its
>> common parameter overrides that from A?
>>
>
Re: Parameters and Process Groups
Posted by Pierre Villard <pi...@gmail.com>.
Hi,
You probably want to read this:
https://bryanbende.com/development/2021/11/08/apache-nifi-1-15-0-parameter-context-inheritance
Thanks,
Pierre
Le ven. 4 févr. 2022 à 13:38, James McMahon <js...@gmail.com> a écrit :
> We have just started to employ parameter contexts rather than environment
> variables. I have read that you can only have one parameter context for a
> process group. We often have a process group that employs other process
> groups in its flow, let's call the parent group A and the called group B.
> For my questions, there is a parameter context associated with the parent
> process group A.
>
> Is there a concept of parameter context inheritance? Is there parameter
> context override? More specifically:
>
> In the case where the internal called process group B has no parameter
> context set, will it inherit the parameter context from the parent process
> group A?
>
> In the case where the internal called process group B has a parameter
> context set that shares no parameter names in common with the context of A,
> is B aware of a context that is A context + B context?
>
> In the case where the internal called process group B has a parameter
> context set that shares one or more parameter names in common with the
> context of A, is B aware of a context that is A context+B context, but its
> common parameter overrides that from A?
>