You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Salvatore Foss <sa...@gmail.com> on 2021/02/19 21:11:10 UTC
Configurable default connection Prioritizer
Hi,
Do you see much value in being able to specify an instance-wide (or
cluster-wide) default prioritizer for all connections that do not have one
manually set?
Along with the the following properties in nifi.properties:
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB
I'd like to see see something like
nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
PriorityAttributePrioritizer
Thoughts? My only concern would be if connection prioritizers have a
noticeable impact on system resources.
Re: Configurable default connection Prioritizer
Posted by Ryan Hendrickson <ry...@gmail.com>.
Hi Mark,
I searched and didn't see a ticket. I created one here:
https://issues.apache.org/jira/browse/NIFI-9974 | Default Prioritizer for
new Relationships
Thanks,
Ryan
On Fri, Apr 15, 2022 at 10:24 AM Mark Bean <ma...@gmail.com> wrote:
> I like the idea. I'm thinking about how it would be implemented in the UI
> (and API). Specifically, I'm comparing this feature of setting the
> prioritizer to setting default connection properties via the configuration
> of a process group. For the default connection properties, updates to the
> process group configuration only change the default settings for any new
> connection created; it does not affect existing connections. This was
> intentional because it seemed heavy-handed to apply such settings to all
> existing connections - especially expiration settings which could result in
> data loss.
>
> However, the recommendation here is for actively changing the prioritizer
> in all existing connections (and potentially nested connections in child
> process groups). I understand the benefit and use case, but it seems the
> two modification styles (new connections or existing connections) would
> easily become confused.
>
> Would a checkbox for "apply prioritizer to all existing connections" be
> appropriate? And, if so, we still need to somehow make it clear that this
> applies to just prioritizer settings. I do not believe we want the other
> connection settings to be recursively applied to existing connections.
>
> Do we want to introduce a new context menu item for process groups,
> "Connection Settings" or something similar?
>
> Is there a JIRA ticket for this proposed new feature?
>
> On Tue, Apr 12, 2022 at 11:48 PM Ryan Hendrickson <
> ryan.andrew.hendrickson@gmail.com> wrote:
>
> > @Bryan - Correct, everything is still per queue, just with that
> convenience
> > feature.
> >
> > Totally agree with @Salvatore too. I hadn't even thought of nested
> process
> > groups.
> >
> > On Tue, Apr 12, 2022 at 10:14 PM Salvatore <sa...@gmail.com>
> > wrote:
> >
> > > #2 would definitely be convenient. Maybe also include an option whether
> > to
> > > recurse down through nested process groups, or just apply to the
> selected
> > > process group.
> > >
> > > On Wed, 13 Apr 2022 at 06:05, Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > > Makes sense. For # 2, it is still per queue with an "Apply All"
> > > > convenience right? Just trying to differentiate with prioritizing
> > > > across all queues.
> > > >
> > > > On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson
> > > > <ry...@gmail.com> wrote:
> > > > >
> > > > > I see two things as particularly useful...
> > > > >
> > > > > 1) Default Prioritizer for new Relationships (Bound to a process
> > > group,
> > > > > similar to how the "Default FlowFile Expiration" can be changed).
> > > > > 2) Applying a prioritizer to an entire Process Group as a one-time
> > > > action.
> > > > >
> > > > > Some background... I'm hand-converting two super-legacy v0.7.3
> > canvases
> > > > to
> > > > > v1.15.3. Part of that is applying flow priorities all over the
> place
> > > in
> > > > > the new system. Probably not a common task, but I could see this
> > > feature
> > > > > being useful for other week-to-week work too.
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende <bb...@gmail.com>
> > wrote:
> > > > >
> > > > > > I think there are two different concepts here... The original
> > > > > > discussion is just about default settings for new connections.
> The
> > > > > > idea in NIFI-6831 is about prioritizing data across multiple
> > queues,
> > > > > > either for all of nifi or within a given process group.
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean <mark.o.bean@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > We experimented with the idea of a custom "Global Prioritizer".
> > One
> > > > of
> > > > > > the
> > > > > > > problems with this approach is that it ran the risk of breaking
> > the
> > > > > > > multi-tenancy philosophy. If there were a truly global
> priority,
> > it
> > > > would
> > > > > > > affect all flows, each may have different priority rules.
> > However,
> > > if
> > > > > > this
> > > > > > > could be applied only at the process group level, it might have
> > > legs.
> > > > > > >
> > > > > > > You can follow the initial approach to such a mechanism in the
> > JIRA
> > > > > > ticket.
> > > > > > > https://issues.apache.org/jira/browse/NIFI-6831
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > > > > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > > > > >
> > > > > > > > I just went to the config button in my process group, hoping
> to
> > > > set all
> > > > > > > > relationships in there to priority first.... Lots of right
> > > > clicking &
> > > > > > > > dragging instead.
> > > > > > > >
> > > > > > > > +1 for an approach like that.
> > > > > > > >
> > > > > > > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <
> joe.witt@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hello
> > > > > > > > >
> > > > > > > > > Certainly the spirit of this is a good idea. Would likely
> > need
> > > > to
> > > > > > > > approach
> > > > > > > > > it at a more flow/process group centric level.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > > > > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > This would be very helpful.
> > > > > > > > > >
> > > > > > > > > > Ryan
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > > > > > > salvatorefoss@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > Do you see much value in being able to specify an
> > > > instance-wide
> > > > > > (or
> > > > > > > > > > > cluster-wide) default prioritizer for all connections
> > that
> > > > do not
> > > > > > > > have
> > > > > > > > > > one
> > > > > > > > > > > manually set?
> > > > > > > > > > >
> > > > > > > > > > > Along with the the following properties in
> > nifi.properties:
> > > > > > > > > > > nifi.queue.backpressure.count=10000
> > > > > > > > > > > nifi.queue.backpressure.size=1 GB
> > > > > > > > > > >
> > > > > > > > > > > I'd like to see see something like
> > > > > > > > > > >
> > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > > > > > > PriorityAttributePrioritizer
> > > > > > > > > > >
> > > > > > > > > > > Thoughts? My only concern would be if connection
> > > prioritizers
> > > > > > have a
> > > > > > > > > > > noticeable impact on system resources.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: Configurable default connection Prioritizer
Posted by Mark Bean <ma...@gmail.com>.
I like the idea. I'm thinking about how it would be implemented in the UI
(and API). Specifically, I'm comparing this feature of setting the
prioritizer to setting default connection properties via the configuration
of a process group. For the default connection properties, updates to the
process group configuration only change the default settings for any new
connection created; it does not affect existing connections. This was
intentional because it seemed heavy-handed to apply such settings to all
existing connections - especially expiration settings which could result in
data loss.
However, the recommendation here is for actively changing the prioritizer
in all existing connections (and potentially nested connections in child
process groups). I understand the benefit and use case, but it seems the
two modification styles (new connections or existing connections) would
easily become confused.
Would a checkbox for "apply prioritizer to all existing connections" be
appropriate? And, if so, we still need to somehow make it clear that this
applies to just prioritizer settings. I do not believe we want the other
connection settings to be recursively applied to existing connections.
Do we want to introduce a new context menu item for process groups,
"Connection Settings" or something similar?
Is there a JIRA ticket for this proposed new feature?
On Tue, Apr 12, 2022 at 11:48 PM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> @Bryan - Correct, everything is still per queue, just with that convenience
> feature.
>
> Totally agree with @Salvatore too. I hadn't even thought of nested process
> groups.
>
> On Tue, Apr 12, 2022 at 10:14 PM Salvatore <sa...@gmail.com>
> wrote:
>
> > #2 would definitely be convenient. Maybe also include an option whether
> to
> > recurse down through nested process groups, or just apply to the selected
> > process group.
> >
> > On Wed, 13 Apr 2022 at 06:05, Bryan Bende <bb...@gmail.com> wrote:
> >
> > > Makes sense. For # 2, it is still per queue with an "Apply All"
> > > convenience right? Just trying to differentiate with prioritizing
> > > across all queues.
> > >
> > > On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson
> > > <ry...@gmail.com> wrote:
> > > >
> > > > I see two things as particularly useful...
> > > >
> > > > 1) Default Prioritizer for new Relationships (Bound to a process
> > group,
> > > > similar to how the "Default FlowFile Expiration" can be changed).
> > > > 2) Applying a prioritizer to an entire Process Group as a one-time
> > > action.
> > > >
> > > > Some background... I'm hand-converting two super-legacy v0.7.3
> canvases
> > > to
> > > > v1.15.3. Part of that is applying flow priorities all over the place
> > in
> > > > the new system. Probably not a common task, but I could see this
> > feature
> > > > being useful for other week-to-week work too.
> > > >
> > > > Ryan
> > > >
> > > > On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende <bb...@gmail.com>
> wrote:
> > > >
> > > > > I think there are two different concepts here... The original
> > > > > discussion is just about default settings for new connections. The
> > > > > idea in NIFI-6831 is about prioritizing data across multiple
> queues,
> > > > > either for all of nifi or within a given process group.
> > > > >
> > > > >
> > > > > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean <ma...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > We experimented with the idea of a custom "Global Prioritizer".
> One
> > > of
> > > > > the
> > > > > > problems with this approach is that it ran the risk of breaking
> the
> > > > > > multi-tenancy philosophy. If there were a truly global priority,
> it
> > > would
> > > > > > affect all flows, each may have different priority rules.
> However,
> > if
> > > > > this
> > > > > > could be applied only at the process group level, it might have
> > legs.
> > > > > >
> > > > > > You can follow the initial approach to such a mechanism in the
> JIRA
> > > > > ticket.
> > > > > > https://issues.apache.org/jira/browse/NIFI-6831
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > > > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > > > >
> > > > > > > I just went to the config button in my process group, hoping to
> > > set all
> > > > > > > relationships in there to priority first.... Lots of right
> > > clicking &
> > > > > > > dragging instead.
> > > > > > >
> > > > > > > +1 for an approach like that.
> > > > > > >
> > > > > > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <jo...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hello
> > > > > > > >
> > > > > > > > Certainly the spirit of this is a good idea. Would likely
> need
> > > to
> > > > > > > approach
> > > > > > > > it at a more flow/process group centric level.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > > > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > This would be very helpful.
> > > > > > > > >
> > > > > > > > > Ryan
> > > > > > > > >
> > > > > > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > > > > > salvatorefoss@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > Do you see much value in being able to specify an
> > > instance-wide
> > > > > (or
> > > > > > > > > > cluster-wide) default prioritizer for all connections
> that
> > > do not
> > > > > > > have
> > > > > > > > > one
> > > > > > > > > > manually set?
> > > > > > > > > >
> > > > > > > > > > Along with the the following properties in
> nifi.properties:
> > > > > > > > > > nifi.queue.backpressure.count=10000
> > > > > > > > > > nifi.queue.backpressure.size=1 GB
> > > > > > > > > >
> > > > > > > > > > I'd like to see see something like
> > > > > > > > > >
> nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > > > > > PriorityAttributePrioritizer
> > > > > > > > > >
> > > > > > > > > > Thoughts? My only concern would be if connection
> > prioritizers
> > > > > have a
> > > > > > > > > > noticeable impact on system resources.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
>
Re: Configurable default connection Prioritizer
Posted by Ryan Hendrickson <ry...@gmail.com>.
@Bryan - Correct, everything is still per queue, just with that convenience
feature.
Totally agree with @Salvatore too. I hadn't even thought of nested process
groups.
On Tue, Apr 12, 2022 at 10:14 PM Salvatore <sa...@gmail.com> wrote:
> #2 would definitely be convenient. Maybe also include an option whether to
> recurse down through nested process groups, or just apply to the selected
> process group.
>
> On Wed, 13 Apr 2022 at 06:05, Bryan Bende <bb...@gmail.com> wrote:
>
> > Makes sense. For # 2, it is still per queue with an "Apply All"
> > convenience right? Just trying to differentiate with prioritizing
> > across all queues.
> >
> > On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson
> > <ry...@gmail.com> wrote:
> > >
> > > I see two things as particularly useful...
> > >
> > > 1) Default Prioritizer for new Relationships (Bound to a process
> group,
> > > similar to how the "Default FlowFile Expiration" can be changed).
> > > 2) Applying a prioritizer to an entire Process Group as a one-time
> > action.
> > >
> > > Some background... I'm hand-converting two super-legacy v0.7.3 canvases
> > to
> > > v1.15.3. Part of that is applying flow priorities all over the place
> in
> > > the new system. Probably not a common task, but I could see this
> feature
> > > being useful for other week-to-week work too.
> > >
> > > Ryan
> > >
> > > On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > > I think there are two different concepts here... The original
> > > > discussion is just about default settings for new connections. The
> > > > idea in NIFI-6831 is about prioritizing data across multiple queues,
> > > > either for all of nifi or within a given process group.
> > > >
> > > >
> > > > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean <ma...@gmail.com>
> > wrote:
> > > > >
> > > > > We experimented with the idea of a custom "Global Prioritizer". One
> > of
> > > > the
> > > > > problems with this approach is that it ran the risk of breaking the
> > > > > multi-tenancy philosophy. If there were a truly global priority, it
> > would
> > > > > affect all flows, each may have different priority rules. However,
> if
> > > > this
> > > > > could be applied only at the process group level, it might have
> legs.
> > > > >
> > > > > You can follow the initial approach to such a mechanism in the JIRA
> > > > ticket.
> > > > > https://issues.apache.org/jira/browse/NIFI-6831
> > > > >
> > > > >
> > > > > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > > >
> > > > > > I just went to the config button in my process group, hoping to
> > set all
> > > > > > relationships in there to priority first.... Lots of right
> > clicking &
> > > > > > dragging instead.
> > > > > >
> > > > > > +1 for an approach like that.
> > > > > >
> > > > > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <jo...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hello
> > > > > > >
> > > > > > > Certainly the spirit of this is a good idea. Would likely need
> > to
> > > > > > approach
> > > > > > > it at a more flow/process group centric level.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > > > > >
> > > > > > > > This would be very helpful.
> > > > > > > >
> > > > > > > > Ryan
> > > > > > > >
> > > > > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > > > > salvatorefoss@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > Do you see much value in being able to specify an
> > instance-wide
> > > > (or
> > > > > > > > > cluster-wide) default prioritizer for all connections that
> > do not
> > > > > > have
> > > > > > > > one
> > > > > > > > > manually set?
> > > > > > > > >
> > > > > > > > > Along with the the following properties in nifi.properties:
> > > > > > > > > nifi.queue.backpressure.count=10000
> > > > > > > > > nifi.queue.backpressure.size=1 GB
> > > > > > > > >
> > > > > > > > > I'd like to see see something like
> > > > > > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > > > > PriorityAttributePrioritizer
> > > > > > > > >
> > > > > > > > > Thoughts? My only concern would be if connection
> prioritizers
> > > > have a
> > > > > > > > > noticeable impact on system resources.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>
Re: Configurable default connection Prioritizer
Posted by Salvatore <sa...@gmail.com>.
#2 would definitely be convenient. Maybe also include an option whether to
recurse down through nested process groups, or just apply to the selected
process group.
On Wed, 13 Apr 2022 at 06:05, Bryan Bende <bb...@gmail.com> wrote:
> Makes sense. For # 2, it is still per queue with an "Apply All"
> convenience right? Just trying to differentiate with prioritizing
> across all queues.
>
> On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson
> <ry...@gmail.com> wrote:
> >
> > I see two things as particularly useful...
> >
> > 1) Default Prioritizer for new Relationships (Bound to a process group,
> > similar to how the "Default FlowFile Expiration" can be changed).
> > 2) Applying a prioritizer to an entire Process Group as a one-time
> action.
> >
> > Some background... I'm hand-converting two super-legacy v0.7.3 canvases
> to
> > v1.15.3. Part of that is applying flow priorities all over the place in
> > the new system. Probably not a common task, but I could see this feature
> > being useful for other week-to-week work too.
> >
> > Ryan
> >
> > On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende <bb...@gmail.com> wrote:
> >
> > > I think there are two different concepts here... The original
> > > discussion is just about default settings for new connections. The
> > > idea in NIFI-6831 is about prioritizing data across multiple queues,
> > > either for all of nifi or within a given process group.
> > >
> > >
> > > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean <ma...@gmail.com>
> wrote:
> > > >
> > > > We experimented with the idea of a custom "Global Prioritizer". One
> of
> > > the
> > > > problems with this approach is that it ran the risk of breaking the
> > > > multi-tenancy philosophy. If there were a truly global priority, it
> would
> > > > affect all flows, each may have different priority rules. However, if
> > > this
> > > > could be applied only at the process group level, it might have legs.
> > > >
> > > > You can follow the initial approach to such a mechanism in the JIRA
> > > ticket.
> > > > https://issues.apache.org/jira/browse/NIFI-6831
> > > >
> > > >
> > > > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > >
> > > > > I just went to the config button in my process group, hoping to
> set all
> > > > > relationships in there to priority first.... Lots of right
> clicking &
> > > > > dragging instead.
> > > > >
> > > > > +1 for an approach like that.
> > > > >
> > > > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <jo...@gmail.com>
> wrote:
> > > > >
> > > > > > Hello
> > > > > >
> > > > > > Certainly the spirit of this is a good idea. Would likely need
> to
> > > > > approach
> > > > > > it at a more flow/process group centric level.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > > > >
> > > > > > > This would be very helpful.
> > > > > > >
> > > > > > > Ryan
> > > > > > >
> > > > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > > > salvatorefoss@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Do you see much value in being able to specify an
> instance-wide
> > > (or
> > > > > > > > cluster-wide) default prioritizer for all connections that
> do not
> > > > > have
> > > > > > > one
> > > > > > > > manually set?
> > > > > > > >
> > > > > > > > Along with the the following properties in nifi.properties:
> > > > > > > > nifi.queue.backpressure.count=10000
> > > > > > > > nifi.queue.backpressure.size=1 GB
> > > > > > > >
> > > > > > > > I'd like to see see something like
> > > > > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > > > PriorityAttributePrioritizer
> > > > > > > >
> > > > > > > > Thoughts? My only concern would be if connection prioritizers
> > > have a
> > > > > > > > noticeable impact on system resources.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
>
Re: Configurable default connection Prioritizer
Posted by Bryan Bende <bb...@gmail.com>.
Makes sense. For # 2, it is still per queue with an "Apply All"
convenience right? Just trying to differentiate with prioritizing
across all queues.
On Tue, Apr 12, 2022 at 3:22 PM Ryan Hendrickson
<ry...@gmail.com> wrote:
>
> I see two things as particularly useful...
>
> 1) Default Prioritizer for new Relationships (Bound to a process group,
> similar to how the "Default FlowFile Expiration" can be changed).
> 2) Applying a prioritizer to an entire Process Group as a one-time action.
>
> Some background... I'm hand-converting two super-legacy v0.7.3 canvases to
> v1.15.3. Part of that is applying flow priorities all over the place in
> the new system. Probably not a common task, but I could see this feature
> being useful for other week-to-week work too.
>
> Ryan
>
> On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende <bb...@gmail.com> wrote:
>
> > I think there are two different concepts here... The original
> > discussion is just about default settings for new connections. The
> > idea in NIFI-6831 is about prioritizing data across multiple queues,
> > either for all of nifi or within a given process group.
> >
> >
> > On Tue, Apr 12, 2022 at 1:13 PM Mark Bean <ma...@gmail.com> wrote:
> > >
> > > We experimented with the idea of a custom "Global Prioritizer". One of
> > the
> > > problems with this approach is that it ran the risk of breaking the
> > > multi-tenancy philosophy. If there were a truly global priority, it would
> > > affect all flows, each may have different priority rules. However, if
> > this
> > > could be applied only at the process group level, it might have legs.
> > >
> > > You can follow the initial approach to such a mechanism in the JIRA
> > ticket.
> > > https://issues.apache.org/jira/browse/NIFI-6831
> > >
> > >
> > > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > > ryan.andrew.hendrickson@gmail.com> wrote:
> > >
> > > > I just went to the config button in my process group, hoping to set all
> > > > relationships in there to priority first.... Lots of right clicking &
> > > > dragging instead.
> > > >
> > > > +1 for an approach like that.
> > > >
> > > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <jo...@gmail.com> wrote:
> > > >
> > > > > Hello
> > > > >
> > > > > Certainly the spirit of this is a good idea. Would likely need to
> > > > approach
> > > > > it at a more flow/process group centric level.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > > >
> > > > > > This would be very helpful.
> > > > > >
> > > > > > Ryan
> > > > > >
> > > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > > salvatorefoss@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Do you see much value in being able to specify an instance-wide
> > (or
> > > > > > > cluster-wide) default prioritizer for all connections that do not
> > > > have
> > > > > > one
> > > > > > > manually set?
> > > > > > >
> > > > > > > Along with the the following properties in nifi.properties:
> > > > > > > nifi.queue.backpressure.count=10000
> > > > > > > nifi.queue.backpressure.size=1 GB
> > > > > > >
> > > > > > > I'd like to see see something like
> > > > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > > PriorityAttributePrioritizer
> > > > > > >
> > > > > > > Thoughts? My only concern would be if connection prioritizers
> > have a
> > > > > > > noticeable impact on system resources.
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
Re: Configurable default connection Prioritizer
Posted by Ryan Hendrickson <ry...@gmail.com>.
I see two things as particularly useful...
1) Default Prioritizer for new Relationships (Bound to a process group,
similar to how the "Default FlowFile Expiration" can be changed).
2) Applying a prioritizer to an entire Process Group as a one-time action.
Some background... I'm hand-converting two super-legacy v0.7.3 canvases to
v1.15.3. Part of that is applying flow priorities all over the place in
the new system. Probably not a common task, but I could see this feature
being useful for other week-to-week work too.
Ryan
On Tue, Apr 12, 2022 at 1:32 PM Bryan Bende <bb...@gmail.com> wrote:
> I think there are two different concepts here... The original
> discussion is just about default settings for new connections. The
> idea in NIFI-6831 is about prioritizing data across multiple queues,
> either for all of nifi or within a given process group.
>
>
> On Tue, Apr 12, 2022 at 1:13 PM Mark Bean <ma...@gmail.com> wrote:
> >
> > We experimented with the idea of a custom "Global Prioritizer". One of
> the
> > problems with this approach is that it ran the risk of breaking the
> > multi-tenancy philosophy. If there were a truly global priority, it would
> > affect all flows, each may have different priority rules. However, if
> this
> > could be applied only at the process group level, it might have legs.
> >
> > You can follow the initial approach to such a mechanism in the JIRA
> ticket.
> > https://issues.apache.org/jira/browse/NIFI-6831
> >
> >
> > On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> > ryan.andrew.hendrickson@gmail.com> wrote:
> >
> > > I just went to the config button in my process group, hoping to set all
> > > relationships in there to priority first.... Lots of right clicking &
> > > dragging instead.
> > >
> > > +1 for an approach like that.
> > >
> > > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <jo...@gmail.com> wrote:
> > >
> > > > Hello
> > > >
> > > > Certainly the spirit of this is a good idea. Would likely need to
> > > approach
> > > > it at a more flow/process group centric level.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > > ryan.andrew.hendrickson@gmail.com> wrote:
> > > >
> > > > > This would be very helpful.
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > > salvatorefoss@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Do you see much value in being able to specify an instance-wide
> (or
> > > > > > cluster-wide) default prioritizer for all connections that do not
> > > have
> > > > > one
> > > > > > manually set?
> > > > > >
> > > > > > Along with the the following properties in nifi.properties:
> > > > > > nifi.queue.backpressure.count=10000
> > > > > > nifi.queue.backpressure.size=1 GB
> > > > > >
> > > > > > I'd like to see see something like
> > > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > > PriorityAttributePrioritizer
> > > > > >
> > > > > > Thoughts? My only concern would be if connection prioritizers
> have a
> > > > > > noticeable impact on system resources.
> > > > > >
> > > > >
> > > >
> > >
>
Re: Configurable default connection Prioritizer
Posted by Bryan Bende <bb...@gmail.com>.
I think there are two different concepts here... The original
discussion is just about default settings for new connections. The
idea in NIFI-6831 is about prioritizing data across multiple queues,
either for all of nifi or within a given process group.
On Tue, Apr 12, 2022 at 1:13 PM Mark Bean <ma...@gmail.com> wrote:
>
> We experimented with the idea of a custom "Global Prioritizer". One of the
> problems with this approach is that it ran the risk of breaking the
> multi-tenancy philosophy. If there were a truly global priority, it would
> affect all flows, each may have different priority rules. However, if this
> could be applied only at the process group level, it might have legs.
>
> You can follow the initial approach to such a mechanism in the JIRA ticket.
> https://issues.apache.org/jira/browse/NIFI-6831
>
>
> On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
> ryan.andrew.hendrickson@gmail.com> wrote:
>
> > I just went to the config button in my process group, hoping to set all
> > relationships in there to priority first.... Lots of right clicking &
> > dragging instead.
> >
> > +1 for an approach like that.
> >
> > On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Hello
> > >
> > > Certainly the spirit of this is a good idea. Would likely need to
> > approach
> > > it at a more flow/process group centric level.
> > >
> > > Thanks
> > >
> > > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > > ryan.andrew.hendrickson@gmail.com> wrote:
> > >
> > > > This would be very helpful.
> > > >
> > > > Ryan
> > > >
> > > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> > salvatorefoss@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Do you see much value in being able to specify an instance-wide (or
> > > > > cluster-wide) default prioritizer for all connections that do not
> > have
> > > > one
> > > > > manually set?
> > > > >
> > > > > Along with the the following properties in nifi.properties:
> > > > > nifi.queue.backpressure.count=10000
> > > > > nifi.queue.backpressure.size=1 GB
> > > > >
> > > > > I'd like to see see something like
> > > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > > PriorityAttributePrioritizer
> > > > >
> > > > > Thoughts? My only concern would be if connection prioritizers have a
> > > > > noticeable impact on system resources.
> > > > >
> > > >
> > >
> >
Re: Configurable default connection Prioritizer
Posted by Mark Bean <ma...@gmail.com>.
We experimented with the idea of a custom "Global Prioritizer". One of the
problems with this approach is that it ran the risk of breaking the
multi-tenancy philosophy. If there were a truly global priority, it would
affect all flows, each may have different priority rules. However, if this
could be applied only at the process group level, it might have legs.
You can follow the initial approach to such a mechanism in the JIRA ticket.
https://issues.apache.org/jira/browse/NIFI-6831
On Tue, Apr 12, 2022 at 12:06 PM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> I just went to the config button in my process group, hoping to set all
> relationships in there to priority first.... Lots of right clicking &
> dragging instead.
>
> +1 for an approach like that.
>
> On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <jo...@gmail.com> wrote:
>
> > Hello
> >
> > Certainly the spirit of this is a good idea. Would likely need to
> approach
> > it at a more flow/process group centric level.
> >
> > Thanks
> >
> > On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> > ryan.andrew.hendrickson@gmail.com> wrote:
> >
> > > This would be very helpful.
> > >
> > > Ryan
> > >
> > > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <
> salvatorefoss@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Do you see much value in being able to specify an instance-wide (or
> > > > cluster-wide) default prioritizer for all connections that do not
> have
> > > one
> > > > manually set?
> > > >
> > > > Along with the the following properties in nifi.properties:
> > > > nifi.queue.backpressure.count=10000
> > > > nifi.queue.backpressure.size=1 GB
> > > >
> > > > I'd like to see see something like
> > > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > > PriorityAttributePrioritizer
> > > >
> > > > Thoughts? My only concern would be if connection prioritizers have a
> > > > noticeable impact on system resources.
> > > >
> > >
> >
>
Re: Configurable default connection Prioritizer
Posted by Ryan Hendrickson <ry...@gmail.com>.
I just went to the config button in my process group, hoping to set all
relationships in there to priority first.... Lots of right clicking &
dragging instead.
+1 for an approach like that.
On Tue, Apr 12, 2022 at 11:44 AM Joe Witt <jo...@gmail.com> wrote:
> Hello
>
> Certainly the spirit of this is a good idea. Would likely need to approach
> it at a more flow/process group centric level.
>
> Thanks
>
> On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
> ryan.andrew.hendrickson@gmail.com> wrote:
>
> > This would be very helpful.
> >
> > Ryan
> >
> > On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <sa...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Do you see much value in being able to specify an instance-wide (or
> > > cluster-wide) default prioritizer for all connections that do not have
> > one
> > > manually set?
> > >
> > > Along with the the following properties in nifi.properties:
> > > nifi.queue.backpressure.count=10000
> > > nifi.queue.backpressure.size=1 GB
> > >
> > > I'd like to see see something like
> > > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > > PriorityAttributePrioritizer
> > >
> > > Thoughts? My only concern would be if connection prioritizers have a
> > > noticeable impact on system resources.
> > >
> >
>
Re: Configurable default connection Prioritizer
Posted by Joe Witt <jo...@gmail.com>.
Hello
Certainly the spirit of this is a good idea. Would likely need to approach
it at a more flow/process group centric level.
Thanks
On Tue, Apr 12, 2022 at 8:34 AM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> This would be very helpful.
>
> Ryan
>
> On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <sa...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Do you see much value in being able to specify an instance-wide (or
> > cluster-wide) default prioritizer for all connections that do not have
> one
> > manually set?
> >
> > Along with the the following properties in nifi.properties:
> > nifi.queue.backpressure.count=10000
> > nifi.queue.backpressure.size=1 GB
> >
> > I'd like to see see something like
> > nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> > PriorityAttributePrioritizer
> >
> > Thoughts? My only concern would be if connection prioritizers have a
> > noticeable impact on system resources.
> >
>
Re: Configurable default connection Prioritizer
Posted by Ryan Hendrickson <ry...@gmail.com>.
This would be very helpful.
Ryan
On Fri, Feb 19, 2021 at 4:51 PM Salvatore Foss <sa...@gmail.com>
wrote:
> Hi,
>
> Do you see much value in being able to specify an instance-wide (or
> cluster-wide) default prioritizer for all connections that do not have one
> manually set?
>
> Along with the the following properties in nifi.properties:
> nifi.queue.backpressure.count=10000
> nifi.queue.backpressure.size=1 GB
>
> I'd like to see see something like
> nifi.queue.prioritizer.default=org.apache.nifi.prioritize.
> PriorityAttributePrioritizer
>
> Thoughts? My only concern would be if connection prioritizers have a
> noticeable impact on system resources.
>