Can Ignite be configured to provide an array like replicated and durable map?

[mfronczyk <fr...@gmail.com>]

(I posted the same question on the users forum but got no answer, so I try
here. I'm sorry if it's not the right place.)

I need a data structure that holds 1 billion of objects. Each object will
consist of 4 integers. This will be used to model a large 2-dimensional
grid. 

The structure is initially empty (or can be full but have objects with all
fields set to 0), but will be filled in with elements inserted at random
places. The data structure is immutable in the sense that each index can be
written only once. This invariant should be guaranteed even when there are
multiple concurrent clients using it - writing to an already written index
should return an error. All writes should be fast and have low latency. 

Another important feature is that the memory layout of the structure matches
the array indexes - element with index n is before the element n + 1. There
will be a need to read 16384 consecutive elements quickly and that memory
layout will be very cache friendly for such operation. 

Elements written to the structure need to be durable. The plan is to
replicate the writes on 2 nodes. Writes to about 1% of the indexes are more
important than others and it should be guaranteed that acknowledged writes
for that indexes survive a complete crash of one node. There should be no
time window during which such writes are lost. But it's OK for the rest of
99% other writes to have a chance of data loss for performance reasons. So
we need async replication with a possibility to enable synchronous
replication for certain puts. 

This all sounds like a replicated, durable array which isn't provided by
Ignite, but I can imagine implementing this as a specialised map/key-value
store, with ints as keys, disabled hashing of keys and the storage allocated
off-heap to guarantee that keys that are close to each other have
consecutive locations in RAM. Ideally, it should also be possible to specify
whether synchronous replication is needed or not for each write operation.
Is it possible to achieve using Apache Ignite?



--
View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp3025.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.