Kaha store

[Rob Davies <ra...@gmail.com>]

Something we need to address for the  Kaha persistent store is  
pluggable strategies for the indexing of data for both the List and  
Map container types.
Currently, an  IndexItem is a fixed size, but can be iterated as a bi- 
directional linked list. The IndexItem holds offsets to the the data  
value and optional data key which is held in rolling data logs. Data  
values/keys are variable in length.
When a Container is loaded, the IndexItem is loaded in to an in- 
memory structure - so for a Map Container a Map will hold the key -  
the value being the IndexItem - so that getting a data value involves  
looking up the IndexItem - then retrieving the data value associated  
with the IndexItem from disk.

This works reasonably well and is fine medium size collections of  
messages (~20-30million with the default heap size for a message  
broker).  But to ensure the Kaha Store can handle indexing larger  
number of messages with a relatively small memory footprint - we are  
obviously going to need something a little more sophisticated.

Apart from a B-Tree - something that I've though about is using  
different types of caching - as the activemq message store really  
uses FIFO queues, a FIFO cache strategy would seem like a good  
strategy? thoughts ?

cheers,

Rob 

Re: Kaha store

[Hiram Chirino <hi...@hiramchirino.com>]

On 7/10/06, Rob Davies <ra...@gmail.com> wrote:
>
>
>
> Something we need to address for the  Kaha persistent store is
> pluggable strategies for the indexing of data for both the List and
> Map container types.


+1

Currently, an  IndexItem is a fixed size, but can be iterated as a bi-
> directional linked list. The IndexItem holds offsets to the the data
> value and optional data key which is held in rolling data logs. Data
> values/keys are variable in length.
> When a Container is loaded, the IndexItem is loaded in to an in-
> memory structure - so for a Map Container a Map will hold the key -
> the value being the IndexItem - so that getting a data value involves
> looking up the IndexItem - then retrieving the data value associated
> with the IndexItem from disk.
>
> This works reasonably well and is fine medium size collections of
> messages (~20-30million with the default heap size for a message
> broker).  But to ensure the Kaha Store can handle indexing larger
> number of messages with a relatively small memory footprint - we are
> obviously going to need something a little more sophisticated.
>
> Apart from a B-Tree - something that I've though about is using
> different types of caching - as the activemq message store really
> uses FIFO queues, a FIFO cache strategy would seem like a good
> strategy? thoughts ?


+1

cheers,
>
> Rob
>



-- 
Regards,
Hiram

Blog: http://hiramchirino.com