Re: Replication & activeMQ, update

[Emmanuel Lecharny <el...@gmail.com>]

Hi,

so we have had a cnvo offline with Kiran early this morning, while most 
of you guys were sleeping :) Let me write down here what we discussed about.

We agreed that using ActiveMQ is totally overkilling. It's like using a 
hammer to kill a fly. We need something lighter, and I thnk the previous 
mails already stated that. What we discussed about this morning was the 
alternative we could use.
Yesterday, I felt a bit depressed about the existing alternative, and as 
usual, my first thought was "let's rewrite this damn journa mechanism 
from scratch". A typical NIH syndrom, something that is often cured by a 
few desperate hours of useless coding, and a good night of sleep.

So this morning, we had a look at KalahDB, the internal Journal system 
used by ActiveMQ. At first, it sound interesting, as it's lightweight, 
and offers what we need : a journal, file backed, with rotation. The 
problem is that we can't easily build a cursor on top of it, as finding 
a CSN from this log requires a full scan of the files. Plus the doco is 
really light, not to mention the total absence of Javadoc and samples... 
(sometime, I feel like our code is one of the most Javadoced code, ever, 
in the OSS world :/ )

Then we agreed that what is needed is to have a CSN index on top of the 
journal, as we want to be able to directly find the position of a 
specific CSN, plus to be able to read all the entries from this CSN in a 
chronological order.

That leads us to the idea to use an Index (the one we define in the 
server) to store the journal. The biggest advantage of using this data 
structure is that it already works, we know how to use it, it's tested, 
it's quite efficient, and it can use any kind of underlying storage 
(JDBM, HBase, LDIF files).

We will test this approach today.

thoughts ?

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com