partially durable queue?

[Noel Grandin <no...@gmail.com>]

Hi

I have a need for a queue that operates purely in-memory until the consumer(s) can't keep up, and only then starts 
spooling to disk.
I'm ok with losing some messages in the event of a hardware or software crash.

Is there such a configuration in RabbitMQ? If not, could one be created with a plugin?

For context, the requirement is slightly weird in that I'm
(a) wanting to push some data back over a flakey WAN
(b) from an underpowered piece of hardware that doesn't like it's little SSD drive being written too hard
:-)

Thanks, Noel Grandin

Re: partially durable queue?

[Clebert Suconic <cl...@gmail.com>]

It will work with artemis. It will start to steam to disk when beyond
configured limit.

On Fri, Jan 26, 2018 at 2:12 PM Noel Grandin <no...@gmail.com> wrote:

> On 26 January 2018 at 20:58, Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > With artemis you can simply disable syncs on producer and you would get
> > similar performance numbers. Usually people turn of persistence to gain
> >
>
> Ah, it's not perf I'm after, think of this as a kind of IOT/sensor device,
> I want to preserve my rather cheap SSD device from wearing out.
>
> I'm producing a number of measurements on 24/7 basis (10-50 messages / sec)
>
> In the common case I should be able to stream this out without ever hitting
> disk, but for backup I need to spool messages if the consumer becomes
> temporarily unreachable.
>
> This is an abnormal use-case, so I'm not surprised it's not readily
> available.
>
> But I figured
> (a) I might get lucky
> (b) if not, it might be worth adding the feature to ActiveMQ myself to
> avoid re-inventing all the other parts
>
-- 
Clebert Suconic

Re: partially durable queue?

[Noel Grandin <no...@gmail.com>]

On 26 January 2018 at 20:58, Clebert Suconic <cl...@gmail.com>
wrote:

> With artemis you can simply disable syncs on producer and you would get
> similar performance numbers. Usually people turn of persistence to gain
>

Ah, it's not perf I'm after, think of this as a kind of IOT/sensor device,
I want to preserve my rather cheap SSD device from wearing out.

I'm producing a number of measurements on 24/7 basis (10-50 messages / sec)

In the common case I should be able to stream this out without ever hitting
disk, but for backup I need to spool messages if the consumer becomes
temporarily unreachable.

This is an abnormal use-case, so I'm not surprised it's not readily
available.

But I figured
(a) I might get lucky
(b) if not, it might be worth adding the feature to ActiveMQ myself to
avoid re-inventing all the other parts

Re: partially durable queue?

[Clebert Suconic <cl...@gmail.com>]

With artemis you can simply disable syncs on producer and you would get
similar performance numbers. Usually people turn of persistence to gain
some performance.  With artemis you can just always persist and increase
perf by batching multiple writes. SyncOnDurable=false on the connection
factory.

On Fri, Jan 26, 2018 at 12:51 PM Noel Grandin <no...@gmail.com> wrote:

> On Fri, 26 Jan 2018 at 19:22, Tim Bain <tb...@alumni.duke.edu> wrote:
>
> >
> > 5.x can also do in-memory-till-full-then-spool-to-disk when using
> > non-persistent messages (which go to the memory store and then to the
> temp
> > store if the memory store is full).
> >
>
> That’s useful info thanks. But non persistent means that the broker will
> throw away the messages on a reboot? Which wouldn’t be good for me.
> I can lose a small window of data, but if the wan disconnects for a while
> and then the machine reboots I need to keep most of the data.
>
>
>
>
> >
>
-- 
Clebert Suconic

Re: partially durable queue?

[Tim Bain <tb...@alumni.duke.edu>]

Sorry this took a while (I've not had much free time to look through the
source code recently) and you may well have already found what you need in
the meantime, but in case you haven't yet, I think you'd want to be looking
in the activemq-kahadb-store project and particularly
org.apache.activemq.store.kahadb.plist.PListImpl or PListStoreImpl. Based
on what you've said, I think you'll be looking to have the PListImpl store
messages in memory without writing to the PageFile until some threshold is
reached, and then begin writing messages into the PageFile at that time.

Since I've not done very much poking in the KahaDB code, it's possible that
I'm off the mark here on where best to implement this, but it's where I'd
start looking if I were you.

Tim

On Mon, Jan 29, 2018 at 3:53 AM, Noel Grandin <no...@gmail.com> wrote:

> Thanks for all the answers. It sounds like I can't quite achieve what I
> want out of the box. (Which is not a surprise)
>
> I'll have a look at the internal code and see how hard it will be to build
> something custom that is basically a combination of a small non-persistent
> in-memory queue, and a larger on-disk persistent queue.
>
> Any ideas where to start looking in the codebase?
>

Re: partially durable queue?

[Noel Grandin <no...@gmail.com>]

Thanks for all the answers. It sounds like I can't quite achieve what I
want out of the box. (Which is not a surprise)

I'll have a look at the internal code and see how hard it will be to build
something custom that is basically a combination of a small non-persistent
in-memory queue, and a larger on-disk persistent queue.

Any ideas where to start looking in the codebase?

Re: partially durable queue?

[Tim Bain <tb...@alumni.duke.edu>]

It'll only throw away unconsumed messages. So a single event like a
whole-house power failure won't lose many messages, because the consumer
will probably be up to date and there won't be any messages on the broker.
You're only at risk of data loss if there are multiple failures separated
by a significant amount of time (the consumer dies, and then substantially
later the broker dies due to something unrelated). It might be that the
odds of that are low enough and the consequences reasonable enough that you
can accept that risk, or maybe not.

Also, keep in mind that if you're keeping data in memory until you fill
memory, your messages are non-persistent and will be lost when the message
broker cycles. That's true for any messaging product configured the way
you've said you want it, so it's not unique to ActiveMQ 5.x. If losing most
or all of your unconsumed messages is a problem, you need to be storing
them on disk even if it wears out your cheap SSDs, or you need to stand up
a central broker on something with an attached spinning disk.

Tim

On Jan 26, 2018 10:51 AM, "Noel Grandin" <no...@gmail.com> wrote:

On Fri, 26 Jan 2018 at 19:22, Tim Bain <tb...@alumni.duke.edu> wrote:

>
> 5.x can also do in-memory-till-full-then-spool-to-disk when using
> non-persistent messages (which go to the memory store and then to the temp
> store if the memory store is full).
>

That’s useful info thanks. But non persistent means that the broker will
throw away the messages on a reboot? Which wouldn’t be good for me.
I can lose a small window of data, but if the wan disconnects for a while
and then the machine reboots I need to keep most of the data.




>

Re: partially durable queue?

[Noel Grandin <no...@gmail.com>]

On Fri, 26 Jan 2018 at 19:22, Tim Bain <tb...@alumni.duke.edu> wrote:

>
> 5.x can also do in-memory-till-full-then-spool-to-disk when using
> non-persistent messages (which go to the memory store and then to the temp
> store if the memory store is full).
>

That’s useful info thanks. But non persistent means that the broker will
throw away the messages on a reboot? Which wouldn’t be good for me.
I can lose a small window of data, but if the wan disconnects for a while
and then the machine reboots I need to keep most of the data.




>

Re: partially durable queue?

[Tim Bain <tb...@alumni.duke.edu>]

This mailing list is for the two ActiveMQ forks: 5.x and Artemis (which is
expected to become 6.x). RabbitMQ and all other message brokers are
unrelated products.

5.x can also do in-memory-till-full-then-spool-to-disk when using
non-persistent messages (which go to the memory store and then to the temp
store if the memory store is full).

Pro tip: spending two minutes Googling products you've never heard of is a
much better use of everyone's time than just replying to say you don't know
what they are.

Tim

On Jan 26, 2018 9:49 AM, "Noel Grandin" <no...@gmail.com> wrote:

> On Fri, 26 Jan 2018 at 17:59, Justin Bertram <jb...@apache.org> wrote:
>
> >
> > If you're asking about ActiveMQ, can you clarify if you're asking about
> the
> > 5.x or Artemis broker?
> >
>
>
> I’ve only just started looking at tech options, so I don’t even know what
> those are. I assume different forks of active mq?
>

Re: partially durable queue?

[Noel Grandin <no...@gmail.com>]

On Fri, 26 Jan 2018 at 17:59, Justin Bertram <jb...@apache.org> wrote:

>
> If you're asking about ActiveMQ, can you clarify if you're asking about the
> 5.x or Artemis broker?
>


I’ve only just started looking at tech options, so I don’t even know what
those are. I assume different forks of active mq?

Re: partially durable queue?

[Justin Bertram <jb...@apache.org>]

Are you asking about RabbitMQ?  If so, the ActiveMQ user list might not be
the best place to get answers about RabbitMQ.

If you're asking about ActiveMQ, can you clarify if you're asking about the
5.x or Artemis broker?


Justin

On Fri, Jan 26, 2018 at 9:15 AM, Noel Grandin <no...@gmail.com> wrote:

>
> Hi
>
> I have a need for a queue that operates purely in-memory until the
> consumer(s) can't keep up, and only then starts spooling to disk.
> I'm ok with losing some messages in the event of a hardware or software
> crash.
>
> Is there such a configuration in RabbitMQ? If not, could one be created
> with a plugin?
>
> For context, the requirement is slightly weird in that I'm
> (a) wanting to push some data back over a flakey WAN
> (b) from an underpowered piece of hardware that doesn't like it's little
> SSD drive being written too hard
> :-)
>
> Thanks, Noel Grandin
>
>

Re: partially durable queue?

[Clebert Suconic <cl...@gmail.com>]

You can do that with Artemis and Paging.. I'm not sure about Rabbit :)

On Fri, Jan 26, 2018 at 10:15 AM, Noel Grandin <no...@gmail.com> wrote:
>
> Hi
>
> I have a need for a queue that operates purely in-memory until the
> consumer(s) can't keep up, and only then starts spooling to disk.
> I'm ok with losing some messages in the event of a hardware or software
> crash.
>
> Is there such a configuration in RabbitMQ? If not, could one be created with
> a plugin?
>
> For context, the requirement is slightly weird in that I'm
> (a) wanting to push some data back over a flakey WAN
> (b) from an underpowered piece of hardware that doesn't like it's little SSD
> drive being written too hard
> :-)
>
> Thanks, Noel Grandin
>



-- 
Clebert Suconic