You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by mfronczyk <fr...@gmail.com> on 2015/09/09 01:28:31 UTC
Can Ignite be configured to provide an array like replicated and
durable map?
I need a data structure that holds 1 billion of objects. Each object will
consist of 4 integers. This will be used to model a large 2-dimensional
grid.
The structure is initially empty (or can be full but have objects with all
fields set to 0), but will be filled in with elements inserted at random
places. The data structure is immutable in the sense that each index can be
written only once. This invariant should be guaranteed even when there are
multiple concurrent clients using it - writing to an already written index
should return an error. All writes should be fast and have low latency.
Another important feature is that the memory layout of the structure matches
the array indexes - element with index n is before the element n + 1. There
will be a need to read 16384 consecutive elements quickly and that memory
layout will be very cache friendly for such operation.
Elements written to the structure need to be durable. The plan is to
replicate the writes on 2 nodes. Writes to about 1% of the indexes are more
important than others and it should be guaranteed that acknowledged writes
for that indexes survive a complete crash of one node. There should be no
time window during which such writes are lost. But it's OK for the rest of
99% other writes to have a chance of data loss for performance reasons. So
we need async replication with a possibility to enable synchronous
replication for certain puts.
This all sounds like a replicated, durable array which isn't provided by
Ignite, but I can imagine implementing this as a specialised map/key-value
store, with ints as keys, disabled hashing of keys and the storage allocated
off-heap to guarantee that keys that are close to each other have
consecutive locations in RAM. Ideally, it should also be possible to specify
whether synchronous replication is needed or not for each write operation.
Is it possible to achieve using Apache Ignite?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Can Ignite be configured to provide an array like replicated
and durable map?
Posted by vkulichenko <va...@gmail.com>.
mfronczyk wrote
> Makes sense. CacheStore needs to clear that transient information after
> writing the data to the persistent store. I'm assuming it doesn't require
> any synchronization with the entry processor in the write-through scenario
> because the key is still locked in CacheStore.
That's right. Actually, clean up is not required, because a copy of the
value will be created each time you read it, so entry processor will not get
the same instance that is stored in cache. BTW, if you want to override this
behavior for performance reasons, set
CacheConfiguration.setCopyOnRead(false) property.
mfronczyk wrote
> How will it work in the write-behind scenario? Are writes blocked during
> persisting in the write-behind mode and it's safe to modify the value
> passed to the store after persisting it?
With write-behind writes to persistence store happen asynchronously in
background, so they don't block any cache operations. If copyOnRead=true
(default value), it should work for you, because the independent copy will
be written to the store. copyOnRead=false should not be used with
write-behind at all, because the value written to store will be likely
updated by that time.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1370.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Can Ignite be configured to provide an array like replicated
and durable map?
Posted by mfronczyk <fr...@gmail.com>.
Makes sense. CacheStore needs to clear that transient information after
writing the data to the persistent store. I'm assuming it doesn't require
any synchronization with the entry processor in the write-through scenario
because the key is still locked in CacheStore.
How will it work in the write-behind scenario? Are writes blocked during
persisting in the write-behind mode and it's safe to modify the value
passed to the store after persisting it?
On Fri, Sep 11, 2015 at 8:29 PM, vkulichenko [via Apache Ignite Users] <
ml-node+s70518n1366h87@n6.nabble.com> wrote:
> mfronczyk wrote
> How will it work with Persistent Store configured with write-through or
> write-behind? Won't the whole value (16384 element array) be persisted
> each
> time a single entry of the array is modified?
>
> The whole entry will be passed to CacheStore.write(..) method and it's up
> to your implementation what is actually persisted to the database. The only
> issue I see here is that you need to know which elements were updated when
> the store is called. I think you should add a transient field into the
> value that will contain this information, update it inside entry processor
> and then use inside the store to optimize persistence.
>
> -Val
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1366.html
> To unsubscribe from Can Ignite be configured to provide an array like
> replicated and durable map?, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1304&code=ZnJvbmN6eWtAZ21haWwuY29tfDEzMDR8LTIwNjU4OTMyNTc=>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Michał Fronczyk
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1367.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Can Ignite be configured to provide an array like replicated
and durable map?
Posted by vkulichenko <va...@gmail.com>.
mfronczyk wrote
> How will it work with Persistent Store configured with write-through or
> write-behind? Won't the whole value (16384 element array) be persisted
> each
> time a single entry of the array is modified?
The whole entry will be passed to CacheStore.write(..) method and it's up to
your implementation what is actually persisted to the database. The only
issue I see here is that you need to know which elements were updated when
the store is called. I think you should add a transient field into the value
that will contain this information, update it inside entry processor and
then use inside the store to optimize persistence.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1366.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Can Ignite be configured to provide an array like replicated
and durable map?
Posted by vkulichenko <va...@gmail.com>.
mfronczyk wrote
> Do I need to use EntryProcessor and invoke if I'll guarantee that all data
> is directly accessible by the client application? My idea is to run Ignite
> embedded in the client, instead of as a remote cluster, to minimise the
> latency, and to have a single backup node to which the data will be
> replicated. Invoke seems like an API to execute operations on the cluster.
I would still recommend to use invoke, because otherwise you will send the
whole updated value to backup(s) which is going to be slow in case you store
all 16384 elements as one entry. Also it looks like you need to configure
cache in replicated mode (see [1]).
[1] https://apacheignite.readme.io/docs/cache-modes#replicated-mode
mfronczyk wrote
> Storing 1 billion elements in separate keys isn't probably the way to go.
> As you said, it'll take much more memory and fetching 16k of objects will
> be slow. If I store 16384 array elements as one cache entry, how will it
> work when a single element is modified? Will it have to replicate the
> whole
> array to the backup node and will it cause writing of the whole array to
> the permanent data store?
It depends on cache atomicity mode (see [2]). In ATOMIC mode the whole value
is sent to backup nodes even if invoke method is used to update it. But in
TRANSACTIONAL mode this never happens and only EntryProcessor instance is
sent across network, so I think you should go with transactional cache. Note
that updates in transactional caches are generally slower that in atomic
caches, but it sounds like you're going to have more reads than updates, so
transactional mode should fit your use case.
[2] https://apacheignite.readme.io/docs/transactions
Let us know if you have more questions.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Can Ignite be configured to provide an array like replicated
and durable map?
Posted by mfronczyk <fr...@gmail.com>.
Do I need to use EntryProcessor and invoke if I'll guarantee that all data
is directly accessible by the client application? My idea is to run Ignite
embedded in the client, instead of as a remote cluster, to minimise the
latency, and to have a single backup node to which the data will be
replicated. Invoke seems like an API to execute operations on the cluster.
Storing 1 billion elements in separate keys isn't probably the way to go.
As you said, it'll take much more memory and fetching 16k of objects will
be slow. If I store 16384 array elements as one cache entry, how will it
work when a single element is modified? Will it have to replicate the whole
array to the backup node and will it cause writing of the whole array to
the permanent data store?
On Thu, Sep 10, 2015 at 4:23 AM, dsetrakyan [via Apache Ignite Users] <
ml-node+s70518n1329h40@n6.nabble.com> wrote:
> mfronczyk wrote
> I need a data structure that holds 1 billion of objects. Each object will
> consist of 4 integers. This will be used to model a large 2-dimensional
> grid.
>
> To keep a data structure this big in memory, you should consider using
> Ignite Off-Heap Memory support [1]. Also make sure that you have enough
> memory on your server to fit the whole data set.
>
> [1] http://apacheignite.gridgain.org/docs/off-heap-memory
>
> mfronczyk wrote
> Another important feature is that the memory layout of the structure
> matches the array indexes - element with index n is before the element n +
> 1. There will be a need to read 16384 consecutive elements quickly and that
> memory layout will be very cache friendly for such operation.
>
> If you remove this consecutive memory layout requirement, then you can
> actually store 1 billion of objects separately, using array index as a key.
> In this case, you can use full IgniteCache API, access each element
> separately, and update different elements concurrently. Depending on your
> requirements, this approach may be easier to use. However, this approach
> will also consume more memory than the approach suggested by Valentin.
>
> mfronczyk wrote
> Elements written to the structure need to be durable. The plan is to
> replicate the writes on 2 nodes. Writes to about 1% of the indexes are more
> important than others and it should be guaranteed that acknowledged writes
> for that indexes survive a complete crash of one node. There should be no
> time window during which such writes are lost. But it's OK for the rest of
> 99% other writes to have a chance of data loss for performance reasons. So
> we need async replication with a possibility to enable synchronous
> replication for certain puts.
>
> Synchronous vs. asynchronous backups can be configured on per-cache level
> via FULL_SYNC and PRIMARY_SYNC modes [2]. So if you really need to separate
> synchronous updates from asynchronous updates, you will have to split your
> data structure into 2 parts and store it in 2 different caches. My
> preference would be that you start off with one cache in FULL_SYNC mode,
> and introduce another one in PRIMARY_SYNC mode only if you run into
> performance issue.
>
> [2]
> http://apacheignite.gridgain.org/v1.3/docs/primary-and-backup-copies#synchronous-and-asynchronous-backups
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1329.html
> To unsubscribe from Can Ignite be configured to provide an array like
> replicated and durable map?, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1304&code=ZnJvbmN6eWtAZ21haWwuY29tfDEzMDR8LTIwNjU4OTMyNTc=>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Michał Fronczyk
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1338.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Can Ignite be configured to provide an array like replicated
and durable map?
Posted by dsetrakyan <ds...@apache.org>.
mfronczyk wrote
> I need a data structure that holds 1 billion of objects. Each object will
> consist of 4 integers. This will be used to model a large 2-dimensional
> grid.
To keep a data structure this big in memory, you should consider using
Ignite Off-Heap Memory support [1]. Also make sure that you have enough
memory on your server to fit the whole data set.
[1] http://apacheignite.gridgain.org/docs/off-heap-memory
mfronczyk wrote
> Another important feature is that the memory layout of the structure
> matches the array indexes - element with index n is before the element n +
> 1. There will be a need to read 16384 consecutive elements quickly and
> that memory layout will be very cache friendly for such operation.
If you remove this consecutive memory layout requirement, then you can
actually store 1 billion of objects separately, using array index as a key.
In this case, you can use full IgniteCache API, access each element
separately, and update different elements concurrently. Depending on your
requirements, this approach may be easier to use. However, this approach
will also consume more memory than the approach suggested by Valentin.
mfronczyk wrote
> Elements written to the structure need to be durable. The plan is to
> replicate the writes on 2 nodes. Writes to about 1% of the indexes are
> more important than others and it should be guaranteed that acknowledged
> writes for that indexes survive a complete crash of one node. There should
> be no time window during which such writes are lost. But it's OK for the
> rest of 99% other writes to have a chance of data loss for performance
> reasons. So we need async replication with a possibility to enable
> synchronous replication for certain puts.
Synchronous vs. asynchronous backups can be configured on per-cache level
via FULL_SYNC and PRIMARY_SYNC modes [2]. So if you really need to separate
synchronous updates from asynchronous updates, you will have to split your
data structure into 2 parts and store it in 2 different caches. My
preference would be that you start off with one cache in FULL_SYNC mode, and
introduce another one in PRIMARY_SYNC mode only if you run into performance
issue.
[2]
http://apacheignite.gridgain.org/v1.3/docs/primary-and-backup-copies#synchronous-and-asynchronous-backups
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1329.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Can Ignite be configured to provide an array like replicated
and durable map?
Posted by vkulichenko <va...@gmail.com>.
Hi,
It looks like you will have to to store 16384 elements that are expected to
be read at once as one cache entry. This is the only way to make sure that
all these elements are stored on one node and in consecutive memory.
To update or read only one element you can use EntryProcessor and
IgniteCache.invoke(..) method (see [1]).
Will this work for you?
[1] https://apacheignite.readme.io/docs/jcache#entryprocessor
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1328.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.