You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Tarou, Kirk" <Ki...@VerizonWireless.com.INVALID> on 2016/08/19 19:14:37 UTC
Re: [E] Re: User-supplied attributes for processors
I think it would be beneficial to add dynamic properties to any processor
for use further down the flow, not in the processor where the properties
are added.
For example, I may have a lot of ŒListSFTP¹ processors that feed into a
single FetchSFTP, then a single PutFile. It would be nice to be able to
add one or more attributes in ListSFTP to tell PutFile where to write the
file or use RouteOnAttribute to send the file to the appropriate processor.
I realize that this can be accomplished by adding an UpdateAttribute
processor, but that would greatly increase the number of processors in my
flow file.
From a cursory scan of the 0.7.0 processors, it looks like
UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText,
HashAttribute and RouteHL7 are the only processors that actually allow you
to add a property, yet all processors have the "+ New Property² widget on
the ³Properties² page. If arbitrary properties can¹t be added, the ³+ New
Property² widget should be hidden.
- Kirk Tarou
On 8/18/16, 5:11 PM, "Matt Burgess" <ma...@gmail.com> wrote:
>Kirk,
>
>The processors have to explicitly know about dynamic properties (and
>their intent) in order to use them appropriately. For a processor like
>ListSFTP, it could be beneficial to have custom attributes (as
>parameters to the SFTP session perhaps?) but the domain knowledge on
>how they'd be used would have to be coded into the processor itself.
>If you have a use case in mind, please feel free to file an
>improvement Jira for ListSFTP to use dynamic properties.
>
>Regards,
>Matt
>
>On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk
><Ki...@verizonwireless.com.invalid> wrote:
>> Is there some reason why Processors, like ListSFTP, don¹t allow custom
>>attributes to be added? Relatedly, why do these processors allow users
>>to add custom attributes & values in the UI even though it always throws
>>the error:
>> Œ[attribute]¹ validated against Œ[value]¹ is invalid because
>>Œ[attribute]¹ is not a supported property
>>
>> - Kirk Tarou
Re: [E] Re: User-supplied attributes for processors
Posted by "Tarou, Kirk" <Ki...@VerizonWireless.com.INVALID>.
I’ve created a pull request that hides the ‘+’ UI element as necessary.
https://github.com/apache/nifi/pull/969
I’ll open a second Jira ticket to start the discussion around adding an
additional tag to add user-supplied tags to flow files.
- Kirk Tarou
On 8/22/16, 6:06 PM, "Tarou, Kirk" <Ki...@VerizonWireless.com> wrote:
>I created ticket a Jira ticket to remove the + icon:
>https://issues.apache.org/jira/browse/NIFI-2629
>
>The existence of a “DynamicProperty” annotation could be used to determine
>whether to show the icon.
>
>I’m making an initial attempt to add a ‘hasDynamicProperty’ field to the
>processor API response.
>
>- Kirk Tarou
>
>
>
>On 8/19/16, 6:41 PM, "Matt Burgess" <ma...@gmail.com> wrote:
>
>>For #1, totally agree that if a processor doesn't support dynamic
>>properties then the UI element could be hidden or disabled.
>>
>>For #2, I like the additional tab idea, it basically enables a decorator
>>pattern on the processor to add attributes without any onus on any part
>>of the flow to use them.
>>
>>Regards,
>>Matt
>>
>>> On Aug 19, 2016, at 8:38 PM, Joe Witt <jo...@gmail.com> wrote:
>>>
>>> Kirk,
>>>
>>> As Matt points out dynamic properties of processors have meaning
>>> specific to those processors. So, we need to be careful to avoid
>>> complicating that.
>>>
>>> You raise two other points there though that I'd like to further
>>>discuss:
>>> 1) Processors that don't really support dynamic properties should not
>>> have the + icon present (or it should be disabled)
>>> - I agree. Can you please raise a JIRA.
>>>
>>> 2) You'd like to be able to add a set of attributes to flow files that
>>> exit a given processor and would like to be able to do this without
>>> adding extra processors.
>>> - I can see how this would be helpful and cleaner in some cases. I
>>> envision that not as processor properties but rather a tab on
>>> processor configuration that lets you tag flow file attributes on flow
>>> files and which supports expression language. We should be really
>>> clear where in the flow file lifecycle that tagging occurs such as on
>>> session commit for example (so you know all other things in that
>>> processor are done). Please raise a JIRA if this is consistent with
>>> your idea.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Fri, Aug 19, 2016 at 12:14 PM, Tarou, Kirk
>>> <Ki...@verizonwireless.com.invalid> wrote:
>>>> I think it would be beneficial to add dynamic properties to any
>>>>processor
>>>> for use further down the flow, not in the processor where the
>>>>properties
>>>> are added.
>>>>
>>>> For example, I may have a lot of ŒListSFTP¹ processors that feed into
>>>>a
>>>> single FetchSFTP, then a single PutFile. It would be nice to be able
>>>>to
>>>> add one or more attributes in ListSFTP to tell PutFile where to write
>>>>the
>>>> file or use RouteOnAttribute to send the file to the appropriate
>>>>processor.
>>>>
>>>> I realize that this can be accomplished by adding an UpdateAttribute
>>>> processor, but that would greatly increase the number of processors in
>>>>my
>>>> flow file.
>>>>
>>>> From a cursory scan of the 0.7.0 processors, it looks like
>>>> UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText,
>>>> HashAttribute and RouteHL7 are the only processors that actually allow
>>>>you
>>>> to add a property, yet all processors have the "+ New Property² widget
>>>>on
>>>> the ³Properties² page. If arbitrary properties can¹t be added, the ³+
>>>>New
>>>> Property² widget should be hidden.
>>>>
>>>> - Kirk Tarou
>>>>
>>>>
>>>>
>>>>> On 8/18/16, 5:11 PM, "Matt Burgess" <ma...@gmail.com> wrote:
>>>>>
>>>>> Kirk,
>>>>>
>>>>> The processors have to explicitly know about dynamic properties (and
>>>>> their intent) in order to use them appropriately. For a processor
>>>>>like
>>>>> ListSFTP, it could be beneficial to have custom attributes (as
>>>>> parameters to the SFTP session perhaps?) but the domain knowledge on
>>>>> how they'd be used would have to be coded into the processor itself.
>>>>> If you have a use case in mind, please feel free to file an
>>>>> improvement Jira for ListSFTP to use dynamic properties.
>>>>>
>>>>> Regards,
>>>>> Matt
>>>>>
>>>>> On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk
>>>>> <Ki...@verizonwireless.com.invalid> wrote:
>>>>>> Is there some reason why Processors, like ListSFTP, don¹t allow
>>>>>>custom
>>>>>> attributes to be added? Relatedly, why do these processors allow
>>>>>>users
>>>>>> to add custom attributes & values in the UI even though it always
>>>>>>throws
>>>>>> the error:
>>>>>> Œ[attribute]¹ validated against Œ[value]¹ is invalid because
>>>>>> Œ[attribute]¹ is not a supported property
>>>>>>
>>>>>> - Kirk Tarou
>>>>
>
Re: [E] Re: User-supplied attributes for processors
Posted by "Tarou, Kirk" <Ki...@VerizonWireless.com.INVALID>.
I created ticket a Jira ticket to remove the + icon:
https://issues.apache.org/jira/browse/NIFI-2629
The existence of a “DynamicProperty” annotation could be used to determine
whether to show the icon.
I’m making an initial attempt to add a ‘hasDynamicProperty’ field to the
processor API response.
- Kirk Tarou
On 8/19/16, 6:41 PM, "Matt Burgess" <ma...@gmail.com> wrote:
>For #1, totally agree that if a processor doesn't support dynamic
>properties then the UI element could be hidden or disabled.
>
>For #2, I like the additional tab idea, it basically enables a decorator
>pattern on the processor to add attributes without any onus on any part
>of the flow to use them.
>
>Regards,
>Matt
>
>> On Aug 19, 2016, at 8:38 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Kirk,
>>
>> As Matt points out dynamic properties of processors have meaning
>> specific to those processors. So, we need to be careful to avoid
>> complicating that.
>>
>> You raise two other points there though that I'd like to further
>>discuss:
>> 1) Processors that don't really support dynamic properties should not
>> have the + icon present (or it should be disabled)
>> - I agree. Can you please raise a JIRA.
>>
>> 2) You'd like to be able to add a set of attributes to flow files that
>> exit a given processor and would like to be able to do this without
>> adding extra processors.
>> - I can see how this would be helpful and cleaner in some cases. I
>> envision that not as processor properties but rather a tab on
>> processor configuration that lets you tag flow file attributes on flow
>> files and which supports expression language. We should be really
>> clear where in the flow file lifecycle that tagging occurs such as on
>> session commit for example (so you know all other things in that
>> processor are done). Please raise a JIRA if this is consistent with
>> your idea.
>>
>> Thanks
>> Joe
>>
>> On Fri, Aug 19, 2016 at 12:14 PM, Tarou, Kirk
>> <Ki...@verizonwireless.com.invalid> wrote:
>>> I think it would be beneficial to add dynamic properties to any
>>>processor
>>> for use further down the flow, not in the processor where the
>>>properties
>>> are added.
>>>
>>> For example, I may have a lot of ŒListSFTP¹ processors that feed into a
>>> single FetchSFTP, then a single PutFile. It would be nice to be able to
>>> add one or more attributes in ListSFTP to tell PutFile where to write
>>>the
>>> file or use RouteOnAttribute to send the file to the appropriate
>>>processor.
>>>
>>> I realize that this can be accomplished by adding an UpdateAttribute
>>> processor, but that would greatly increase the number of processors in
>>>my
>>> flow file.
>>>
>>> From a cursory scan of the 0.7.0 processors, it looks like
>>> UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText,
>>> HashAttribute and RouteHL7 are the only processors that actually allow
>>>you
>>> to add a property, yet all processors have the "+ New Property² widget
>>>on
>>> the ³Properties² page. If arbitrary properties can¹t be added, the ³+
>>>New
>>> Property² widget should be hidden.
>>>
>>> - Kirk Tarou
>>>
>>>
>>>
>>>> On 8/18/16, 5:11 PM, "Matt Burgess" <ma...@gmail.com> wrote:
>>>>
>>>> Kirk,
>>>>
>>>> The processors have to explicitly know about dynamic properties (and
>>>> their intent) in order to use them appropriately. For a processor like
>>>> ListSFTP, it could be beneficial to have custom attributes (as
>>>> parameters to the SFTP session perhaps?) but the domain knowledge on
>>>> how they'd be used would have to be coded into the processor itself.
>>>> If you have a use case in mind, please feel free to file an
>>>> improvement Jira for ListSFTP to use dynamic properties.
>>>>
>>>> Regards,
>>>> Matt
>>>>
>>>> On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk
>>>> <Ki...@verizonwireless.com.invalid> wrote:
>>>>> Is there some reason why Processors, like ListSFTP, don¹t allow
>>>>>custom
>>>>> attributes to be added? Relatedly, why do these processors allow
>>>>>users
>>>>> to add custom attributes & values in the UI even though it always
>>>>>throws
>>>>> the error:
>>>>> Œ[attribute]¹ validated against Œ[value]¹ is invalid because
>>>>> Œ[attribute]¹ is not a supported property
>>>>>
>>>>> - Kirk Tarou
>>>
Re: [E] Re: User-supplied attributes for processors
Posted by Matt Burgess <ma...@gmail.com>.
For #1, totally agree that if a processor doesn't support dynamic properties then the UI element could be hidden or disabled.
For #2, I like the additional tab idea, it basically enables a decorator pattern on the processor to add attributes without any onus on any part of the flow to use them.
Regards,
Matt
> On Aug 19, 2016, at 8:38 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Kirk,
>
> As Matt points out dynamic properties of processors have meaning
> specific to those processors. So, we need to be careful to avoid
> complicating that.
>
> You raise two other points there though that I'd like to further discuss:
> 1) Processors that don't really support dynamic properties should not
> have the + icon present (or it should be disabled)
> - I agree. Can you please raise a JIRA.
>
> 2) You'd like to be able to add a set of attributes to flow files that
> exit a given processor and would like to be able to do this without
> adding extra processors.
> - I can see how this would be helpful and cleaner in some cases. I
> envision that not as processor properties but rather a tab on
> processor configuration that lets you tag flow file attributes on flow
> files and which supports expression language. We should be really
> clear where in the flow file lifecycle that tagging occurs such as on
> session commit for example (so you know all other things in that
> processor are done). Please raise a JIRA if this is consistent with
> your idea.
>
> Thanks
> Joe
>
> On Fri, Aug 19, 2016 at 12:14 PM, Tarou, Kirk
> <Ki...@verizonwireless.com.invalid> wrote:
>> I think it would be beneficial to add dynamic properties to any processor
>> for use further down the flow, not in the processor where the properties
>> are added.
>>
>> For example, I may have a lot of ŒListSFTP¹ processors that feed into a
>> single FetchSFTP, then a single PutFile. It would be nice to be able to
>> add one or more attributes in ListSFTP to tell PutFile where to write the
>> file or use RouteOnAttribute to send the file to the appropriate processor.
>>
>> I realize that this can be accomplished by adding an UpdateAttribute
>> processor, but that would greatly increase the number of processors in my
>> flow file.
>>
>> From a cursory scan of the 0.7.0 processors, it looks like
>> UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText,
>> HashAttribute and RouteHL7 are the only processors that actually allow you
>> to add a property, yet all processors have the "+ New Property² widget on
>> the ³Properties² page. If arbitrary properties can¹t be added, the ³+ New
>> Property² widget should be hidden.
>>
>> - Kirk Tarou
>>
>>
>>
>>> On 8/18/16, 5:11 PM, "Matt Burgess" <ma...@gmail.com> wrote:
>>>
>>> Kirk,
>>>
>>> The processors have to explicitly know about dynamic properties (and
>>> their intent) in order to use them appropriately. For a processor like
>>> ListSFTP, it could be beneficial to have custom attributes (as
>>> parameters to the SFTP session perhaps?) but the domain knowledge on
>>> how they'd be used would have to be coded into the processor itself.
>>> If you have a use case in mind, please feel free to file an
>>> improvement Jira for ListSFTP to use dynamic properties.
>>>
>>> Regards,
>>> Matt
>>>
>>> On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk
>>> <Ki...@verizonwireless.com.invalid> wrote:
>>>> Is there some reason why Processors, like ListSFTP, don¹t allow custom
>>>> attributes to be added? Relatedly, why do these processors allow users
>>>> to add custom attributes & values in the UI even though it always throws
>>>> the error:
>>>> Œ[attribute]¹ validated against Œ[value]¹ is invalid because
>>>> Œ[attribute]¹ is not a supported property
>>>>
>>>> - Kirk Tarou
>>
Re: [E] Re: User-supplied attributes for processors
Posted by Joe Witt <jo...@gmail.com>.
Kirk,
As Matt points out dynamic properties of processors have meaning
specific to those processors. So, we need to be careful to avoid
complicating that.
You raise two other points there though that I'd like to further discuss:
1) Processors that don't really support dynamic properties should not
have the + icon present (or it should be disabled)
- I agree. Can you please raise a JIRA.
2) You'd like to be able to add a set of attributes to flow files that
exit a given processor and would like to be able to do this without
adding extra processors.
- I can see how this would be helpful and cleaner in some cases. I
envision that not as processor properties but rather a tab on
processor configuration that lets you tag flow file attributes on flow
files and which supports expression language. We should be really
clear where in the flow file lifecycle that tagging occurs such as on
session commit for example (so you know all other things in that
processor are done). Please raise a JIRA if this is consistent with
your idea.
Thanks
Joe
On Fri, Aug 19, 2016 at 12:14 PM, Tarou, Kirk
<Ki...@verizonwireless.com.invalid> wrote:
> I think it would be beneficial to add dynamic properties to any processor
> for use further down the flow, not in the processor where the properties
> are added.
>
> For example, I may have a lot of ŒListSFTP¹ processors that feed into a
> single FetchSFTP, then a single PutFile. It would be nice to be able to
> add one or more attributes in ListSFTP to tell PutFile where to write the
> file or use RouteOnAttribute to send the file to the appropriate processor.
>
> I realize that this can be accomplished by adding an UpdateAttribute
> processor, but that would greatly increase the number of processors in my
> flow file.
>
> From a cursory scan of the 0.7.0 processors, it looks like
> UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText,
> HashAttribute and RouteHL7 are the only processors that actually allow you
> to add a property, yet all processors have the "+ New Property² widget on
> the ³Properties² page. If arbitrary properties can¹t be added, the ³+ New
> Property² widget should be hidden.
>
> - Kirk Tarou
>
>
>
> On 8/18/16, 5:11 PM, "Matt Burgess" <ma...@gmail.com> wrote:
>
>>Kirk,
>>
>>The processors have to explicitly know about dynamic properties (and
>>their intent) in order to use them appropriately. For a processor like
>>ListSFTP, it could be beneficial to have custom attributes (as
>>parameters to the SFTP session perhaps?) but the domain knowledge on
>>how they'd be used would have to be coded into the processor itself.
>>If you have a use case in mind, please feel free to file an
>>improvement Jira for ListSFTP to use dynamic properties.
>>
>>Regards,
>>Matt
>>
>>On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk
>><Ki...@verizonwireless.com.invalid> wrote:
>>> Is there some reason why Processors, like ListSFTP, don¹t allow custom
>>>attributes to be added? Relatedly, why do these processors allow users
>>>to add custom attributes & values in the UI even though it always throws
>>>the error:
>>> Œ[attribute]¹ validated against Œ[value]¹ is invalid because
>>>Œ[attribute]¹ is not a supported property
>>>
>>> - Kirk Tarou
>