File-based cursors, temporary files and data safety

[Gilles Harloux <gi...@gmail.com>]

Hi,

I am designing a system around an embedded broker using file-based
cursors. I am trying to make sure I understand the way  file-based
cursors work: Am I right in thinking that temporary files is
functionally the same as memory with respect to transactions and data
security? If the system was to crash and it somehow lost or corrupted
temporary files, would it be possible to restart and get those
messages (those that ended up in the temporary files) back from the
message store?

TIA,

Re: File-based cursors, temporary files and data safety

[Gilles Harloux <gi...@gmail.com>]

Thanks for your reply. Yes, I am in fact handling persistent messages,
so it seems it works the way I understood it.

About file-based cursors use: My system uses (in the worst case) a
large number of queues, each with consumers that (in the worst case)
wants a (in the worst case) large volume of messages to consume at
once. Thankfully, the regular case isn't that extreme, with smaller
messages in not-that-fast bursts in not-that-many queues, So I am
trying to build it so that the normal case is handled decently and the
worst case maybe slows down, but doesn't crash (obviously).

At first, I didn't do any specific configuration. What would happen
when I'd stress the system is that it'd exceed the JVM/process/machine
overall memory capacity, thus crashing the process. So then I tried to
limit the broker global memory use, but I'd get deadlocks: Each queue
would fill partially up until the global memory limit gets reached,
then the broker wouldn't accept messages until some get consumed, but
consumption happens only if enough messages get in a queue... So I
configured per-queue memory limits, but from what I saw the limit has
to be high enough to encompass a transaction size, and that would be
too much for the available capacity. So then I switched to file-based
cursors. They allow me to configure memory limits high enough to
handle a transaction, but at the same time not to use up to much
memory by offloading data to disk if it's too big.

So basically, what I am doing is configure a big per-queue memory
limit with a small cursorMemoryHighWaterMark. Memory consumption stays
under control, but lots of parallel big transactions can still happen
(with the drawback that sometimes, big data touches the disk when it
could have been handled in-memory).

 I am all ears if there's a way to do that any better.

On Mon, Oct 15, 2012 at 3:39 PM, Gary Tully <ga...@gmail.com> wrote:
> yes. on a restart that is what happens, so long as the messages are
> persistent. The file cursor is a pending message cursor, so the
> messages are pending for a consumer. if the consumer goes away or the
> broker restarts, the messages are still pending from the store and are
> replayed through the file cursor, which can impact startup time.
> Typically using the default store based cursor is sufficient. The
> store cursor will use memory to cache pending messages as much as it
> can.
>
> On 11 October 2012 15:10, Gilles Harloux <gi...@gmail.com> wrote:
>> Hi,
>>
>> I am designing a system around an embedded broker using file-based
>> cursors. I am trying to make sure I understand the way  file-based
>> cursors work: Am I right in thinking that temporary files is
>> functionally the same as memory with respect to transactions and data
>> security? If the system was to crash and it somehow lost or corrupted
>> temporary files, would it be possible to restart and get those
>> messages (those that ended up in the temporary files) back from the
>> message store?
>>
>> TIA,
>
>
>
> --
> http://redhat.com
> http://blog.garytully.com

Re: File-based cursors, temporary files and data safety

[Gary Tully <ga...@gmail.com>]

yes. on a restart that is what happens, so long as the messages are
persistent. The file cursor is a pending message cursor, so the
messages are pending for a consumer. if the consumer goes away or the
broker restarts, the messages are still pending from the store and are
replayed through the file cursor, which can impact startup time.
Typically using the default store based cursor is sufficient. The
store cursor will use memory to cache pending messages as much as it
can.

On 11 October 2012 15:10, Gilles Harloux <gi...@gmail.com> wrote:
> Hi,
>
> I am designing a system around an embedded broker using file-based
> cursors. I am trying to make sure I understand the way  file-based
> cursors work: Am I right in thinking that temporary files is
> functionally the same as memory with respect to transactions and data
> security? If the system was to crash and it somehow lost or corrupted
> temporary files, would it be possible to restart and get those
> messages (those that ended up in the temporary files) back from the
> message store?
>
> TIA,



-- 
http://redhat.com
http://blog.garytully.com