how to edit queue content

[scott <tc...@gmail.com>]

Hi community,

I've got a question about a feature I would find useful. I've been 
setting up a lot of new flows and testing various configurations, and I 
thought it would be really useful if I could edit the content of queues. 
For example, I can examine each file in the queue, then decide I want to 
keep the second one and the third one, then remove the rest before 
resuming my flow testing. I know I can delete all files, but is there a 
way to have more control over the queue content? Could I delete a 
specific file, or change the order of the queue?

Thanks for your time,

Scott

Re: how to edit queue content

[Andrew Grande <ap...@gmail.com>]

How is it going to work with e.g. 20GB of events in the queue? I'd be
careful, as requirements blow up into a full db with indexes, search, and a
UI on top. If one wanted to filter events, wouldn't a standard processor do
the job better?

Andrew

On Tue, Mar 27, 2018, 12:11 AM Joe Witt <jo...@gmail.com> wrote:

> Scott
> Yep definitely something we've talked about [1].  We've not pursued it
> directly as of yet since it is indeed a queue and we're just letting
> you peak into it.  We dont have facilities built in to really alter
> the queue in a particular position.  Also, the complexity comes in
> when folks want to have paging/selection of various items down the
> list/etc..  (but it isn't a list - its a queue).
>
> If you could bound the range of what you'd expect to be able to do
> that would probably help constrain into something reasonably
> implemented.
>
> Thanks
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Interactive+Queue+Management
>
> On Tue, Mar 27, 2018 at 12:06 AM, scott <tc...@gmail.com> wrote:
> > Hi community,
> >
> > I've got a question about a feature I would find useful. I've been
> setting
> > up a lot of new flows and testing various configurations, and I thought
> it
> > would be really useful if I could edit the content of queues. For
> example, I
> > can examine each file in the queue, then decide I want to keep the second
> > one and the third one, then remove the rest before resuming my flow
> testing.
> > I know I can delete all files, but is there a way to have more control
> over
> > the queue content? Could I delete a specific file, or change the order of
> > the queue?
> >
> > Thanks for your time,
> >
> > Scott
> >
>

Re: how to edit queue content

[Joe Witt <jo...@gmail.com>]

Scott
Yep definitely something we've talked about [1].  We've not pursued it
directly as of yet since it is indeed a queue and we're just letting
you peak into it.  We dont have facilities built in to really alter
the queue in a particular position.  Also, the complexity comes in
when folks want to have paging/selection of various items down the
list/etc..  (but it isn't a list - its a queue).

If you could bound the range of what you'd expect to be able to do
that would probably help constrain into something reasonably
implemented.

Thanks

[1] https://cwiki.apache.org/confluence/display/NIFI/Interactive+Queue+Management

On Tue, Mar 27, 2018 at 12:06 AM, scott <tc...@gmail.com> wrote:
> Hi community,
>
> I've got a question about a feature I would find useful. I've been setting
> up a lot of new flows and testing various configurations, and I thought it
> would be really useful if I could edit the content of queues. For example, I
> can examine each file in the queue, then decide I want to keep the second
> one and the third one, then remove the rest before resuming my flow testing.
> I know I can delete all files, but is there a way to have more control over
> the queue content? Could I delete a specific file, or change the order of
> the queue?
>
> Thanks for your time,
>
> Scott
>

Re: how to edit queue content

[Mike Thomsen <mi...@gmail.com>]

If you know one of the supported scripting languages, you can probably do
some of that with ExecuteScript. For example, if you wanted to drop every
other flowfile in a block of 100, it'd be like this:

def flowfiles = session.get(100) // Get up to 100
int index = 1
flowfiles?.each { flowFile ->
if (index % 2 == 0) {
session.remove(flowFile)
} else {
session.transfer(flowFile, REL_SUCCESS)
}
index++
}

On Tue, Mar 27, 2018 at 12:06 AM, scott <tc...@gmail.com> wrote:

> Hi community,
>
> I've got a question about a feature I would find useful. I've been setting
> up a lot of new flows and testing various configurations, and I thought it
> would be really useful if I could edit the content of queues. For example,
> I can examine each file in the queue, then decide I want to keep the second
> one and the third one, then remove the rest before resuming my flow
> testing. I know I can delete all files, but is there a way to have more
> control over the queue content? Could I delete a specific file, or change
> the order of the queue?
>
> Thanks for your time,
>
> Scott
>
>