question on expiry of paged messages

["Dondorp, Erwin" <er...@cgi.com>]

Hello!

While testing, we produced a queue in Artemis with over 1Gb of data in millions of messages. But there was never any consumer.  All messages had a TTL of one hour.
By now the messages have expired, but the queue browser still shows all/most messages.
These messages have been paged-out, but expiry happens only for messages that are in-memory of the broker (and just before delivery, but these messages will not be delivered).
Of course it is simple to manually drop the queue and then the problem is gone. But so-far we were able to use autocreate+autodelete+expiry+maxexpiry to keep a clean system.
This specific case seems to avoid that...

My question:
Is there any trick (setting) to get the paged-out messages physically expired (i.e. removed) without needing client requests?

Thx!
Erwin

Re: question on expiry of paged messages

[Clebert Suconic <cl...@gmail.com>]

Perhaps you had one (or a few) messages not expired on the queue... on
that case the system will only immediately issue a depage / scan if
all the references in memory have been expired.. on which case not
being a bug... as we can't depage the whole 1G or read the whole 1G in
order to eliminate references.. .you would need to first expire
everything in memory, or consume the ones without expiry.

On Wed, Mar 31, 2021 at 5:56 AM Gary Tully <ga...@gmail.com> wrote:
>
> I see the related jira. some further comments on there
> https://issues.apache.org/jira/browse/ARTEMIS-3214?focusedCommentId=17312236&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312236
>
> On Wed, 31 Mar 2021 at 00:52, Clebert Suconic <cl...@gmail.com> wrote:
> >
> > Just a heads up.
> >
> >
> > We will still need to expire messages that are in memory.  But what I plan
> > to do is to optimize things a bit to it keeps depagjng and scanning as it’s
> > empty.
> >
> >
> > On Tue, Mar 30, 2021 at 6:30 PM Clebert Suconic <cl...@gmail.com>
> > wrote:
> >
> > > Perhaps we could optimize such use case.. issue a dapage an an
> > > immediate scan if all references in memory are expired.
> > >
> > > Do you want to create a JIRA for that?
> > >
> > > On Tue, Mar 30, 2021 at 6:18 PM Clebert Suconic
> > > <cl...@gmail.com> wrote:
> > > >
> > > > All the messages had expiry ?
> > > >
> > > > My expectation was that the system would depare more messages once they
> > > expired through the scanner.
> > > >
> > > >
> > > > But perhaps that’s only happening every scan period milliseconds ?
> > > >
> > > > On Tue, Mar 30, 2021 at 5:24 PM Dondorp, Erwin <er...@cgi.com>
> > > wrote:
> > > >>
> > > >> Hello!
> > > >>
> > > >> While testing, we produced a queue in Artemis with over 1Gb of data in
> > > millions of messages. But there was never any consumer.  All messages had a
> > > TTL of one hour.
> > > >> By now the messages have expired, but the queue browser still shows
> > > all/most messages.
> > > >> These messages have been paged-out, but expiry happens only for
> > > messages that are in-memory of the broker (and just before delivery, but
> > > these messages will not be delivered).
> > > >> Of course it is simple to manually drop the queue and then the problem
> > > is gone. But so-far we were able to use
> > > autocreate+autodelete+expiry+maxexpiry to keep a clean system.
> > > >> This specific case seems to avoid that...
> > > >>
> > > >> My question:
> > > >> Is there any trick (setting) to get the paged-out messages physically
> > > expired (i.e. removed) without needing client requests?
> > > >>
> > > >> Thx!
> > > >> Erwin
> > > >
> > > > --
> > > > Clebert Suconic
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > >
> > --
> > Clebert Suconic



-- 
Clebert Suconic

Re: question on expiry of paged messages

[Gary Tully <ga...@gmail.com>]

I see the related jira. some further comments on there
https://issues.apache.org/jira/browse/ARTEMIS-3214?focusedCommentId=17312236&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312236

On Wed, 31 Mar 2021 at 00:52, Clebert Suconic <cl...@gmail.com> wrote:
>
> Just a heads up.
>
>
> We will still need to expire messages that are in memory.  But what I plan
> to do is to optimize things a bit to it keeps depagjng and scanning as it’s
> empty.
>
>
> On Tue, Mar 30, 2021 at 6:30 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > Perhaps we could optimize such use case.. issue a dapage an an
> > immediate scan if all references in memory are expired.
> >
> > Do you want to create a JIRA for that?
> >
> > On Tue, Mar 30, 2021 at 6:18 PM Clebert Suconic
> > <cl...@gmail.com> wrote:
> > >
> > > All the messages had expiry ?
> > >
> > > My expectation was that the system would depare more messages once they
> > expired through the scanner.
> > >
> > >
> > > But perhaps that’s only happening every scan period milliseconds ?
> > >
> > > On Tue, Mar 30, 2021 at 5:24 PM Dondorp, Erwin <er...@cgi.com>
> > wrote:
> > >>
> > >> Hello!
> > >>
> > >> While testing, we produced a queue in Artemis with over 1Gb of data in
> > millions of messages. But there was never any consumer.  All messages had a
> > TTL of one hour.
> > >> By now the messages have expired, but the queue browser still shows
> > all/most messages.
> > >> These messages have been paged-out, but expiry happens only for
> > messages that are in-memory of the broker (and just before delivery, but
> > these messages will not be delivered).
> > >> Of course it is simple to manually drop the queue and then the problem
> > is gone. But so-far we were able to use
> > autocreate+autodelete+expiry+maxexpiry to keep a clean system.
> > >> This specific case seems to avoid that...
> > >>
> > >> My question:
> > >> Is there any trick (setting) to get the paged-out messages physically
> > expired (i.e. removed) without needing client requests?
> > >>
> > >> Thx!
> > >> Erwin
> > >
> > > --
> > > Clebert Suconic
> >
> >
> >
> > --
> > Clebert Suconic
> >
> --
> Clebert Suconic

Re: question on expiry of paged messages

[Clebert Suconic <cl...@gmail.com>]

Just a heads up.


We will still need to expire messages that are in memory.  But what I plan
to do is to optimize things a bit to it keeps depagjng and scanning as it’s
empty.


On Tue, Mar 30, 2021 at 6:30 PM Clebert Suconic <cl...@gmail.com>
wrote:

> Perhaps we could optimize such use case.. issue a dapage an an
> immediate scan if all references in memory are expired.
>
> Do you want to create a JIRA for that?
>
> On Tue, Mar 30, 2021 at 6:18 PM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > All the messages had expiry ?
> >
> > My expectation was that the system would depare more messages once they
> expired through the scanner.
> >
> >
> > But perhaps that’s only happening every scan period milliseconds ?
> >
> > On Tue, Mar 30, 2021 at 5:24 PM Dondorp, Erwin <er...@cgi.com>
> wrote:
> >>
> >> Hello!
> >>
> >> While testing, we produced a queue in Artemis with over 1Gb of data in
> millions of messages. But there was never any consumer.  All messages had a
> TTL of one hour.
> >> By now the messages have expired, but the queue browser still shows
> all/most messages.
> >> These messages have been paged-out, but expiry happens only for
> messages that are in-memory of the broker (and just before delivery, but
> these messages will not be delivered).
> >> Of course it is simple to manually drop the queue and then the problem
> is gone. But so-far we were able to use
> autocreate+autodelete+expiry+maxexpiry to keep a clean system.
> >> This specific case seems to avoid that...
> >>
> >> My question:
> >> Is there any trick (setting) to get the paged-out messages physically
> expired (i.e. removed) without needing client requests?
> >>
> >> Thx!
> >> Erwin
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
-- 
Clebert Suconic

Re: question on expiry of paged messages

[Clebert Suconic <cl...@gmail.com>]

Perhaps we could optimize such use case.. issue a dapage an an
immediate scan if all references in memory are expired.

Do you want to create a JIRA for that?

On Tue, Mar 30, 2021 at 6:18 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> All the messages had expiry ?
>
> My expectation was that the system would depare more messages once they expired through the scanner.
>
>
> But perhaps that’s only happening every scan period milliseconds ?
>
> On Tue, Mar 30, 2021 at 5:24 PM Dondorp, Erwin <er...@cgi.com> wrote:
>>
>> Hello!
>>
>> While testing, we produced a queue in Artemis with over 1Gb of data in millions of messages. But there was never any consumer.  All messages had a TTL of one hour.
>> By now the messages have expired, but the queue browser still shows all/most messages.
>> These messages have been paged-out, but expiry happens only for messages that are in-memory of the broker (and just before delivery, but these messages will not be delivered).
>> Of course it is simple to manually drop the queue and then the problem is gone. But so-far we were able to use autocreate+autodelete+expiry+maxexpiry to keep a clean system.
>> This specific case seems to avoid that...
>>
>> My question:
>> Is there any trick (setting) to get the paged-out messages physically expired (i.e. removed) without needing client requests?
>>
>> Thx!
>> Erwin
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: question on expiry of paged messages

[Clebert Suconic <cl...@gmail.com>]

All the messages had expiry ?

My expectation was that the system would depare more messages once they
expired through the scanner.


But perhaps that’s only happening every scan period milliseconds ?

On Tue, Mar 30, 2021 at 5:24 PM Dondorp, Erwin <er...@cgi.com>
wrote:

> Hello!
>
> While testing, we produced a queue in Artemis with over 1Gb of data in
> millions of messages. But there was never any consumer.  All messages had a
> TTL of one hour.
> By now the messages have expired, but the queue browser still shows
> all/most messages.
> These messages have been paged-out, but expiry happens only for messages
> that are in-memory of the broker (and just before delivery, but these
> messages will not be delivered).
> Of course it is simple to manually drop the queue and then the problem is
> gone. But so-far we were able to use autocreate+autodelete+expiry+maxexpiry
> to keep a clean system.
> This specific case seems to avoid that...
>
> My question:
> Is there any trick (setting) to get the paged-out messages physically
> expired (i.e. removed) without needing client requests?
>
> Thx!
> Erwin
>
-- 
Clebert Suconic