You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Johannes Lichtenberger <jo...@unitedplanet.com> on 2019/08/06 12:46:55 UTC
Ignite Partitioned Cache / Use for an in-memory transaction-log
Hi,
can I somehow query on which cluster-nodes a partitioned cache stores
values for specific keys? I might want to use a cache for replicating an
in-memory transaction log (which does not have to be persisted) to
replicate a document of a NoSQL document store to a few nodes.
Thus, instead of a local cache I'd simply write into an ignite cache and
then would like to query which cluster-nodes have stored a specific key
(for instance the document name). So for instance I could tell a load
balancer for reads to read documents from one of the backup replicas.
During a transaction commit I would also need this information to know
where to send an event to... to commit the transaction. And somehow I'd
love to wait for a quorum of nodes if the transaction really has
committed or it needs to roll back.
kind regards
Johannes
Re: Ignite Partitioned Cache / Use for an in-memory transaction-log
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
You can have any number of backups, but you can either update all of them
in sync, or primary only. You can't have a subset of backups updated,
despite such possibility being discussed.
Regards,
--
Ilya Kasnacheev
ср, 7 авг. 2019 г. в 13:29, Johannes Lichtenberger <
johannes.lichtenberger@unitedplanet.com>:
> Hi,
>
> if I want to replicate data from one node to N-nodes (that is only a
> subset of nodes in the whole cluster), could I also specify that not only
> the primary is updated in synchronous mode, but some of the backup nodes,
> too?
>
> I also would like to somehow guarantee read-your-own-writes semantics. My
> idea is for replicating a storage / database system to have a single writer
> through which all writes are propagated.
>
> I'd probably use something like oldest node is the master (which can be
> done using Apache Ignite).
>
> Then let's say someone decides to create a database and a first resource.
> I'd have to send an event round robin to create the resource on let's say
> three of 10 cluster nodes. Maybe I could use a distributed semaphore which
> just accepts 3-times acquiring the semaphore (name is database + resource
> concatenated). Once the initial bootstrapping of creating directories and
> files happened on all 3 nodes a write-transaction can be opened, which
> writes into a distributed cache. However as I'm thinking about it the cache
> for this resource must be shared on only the same 3 cluster-nodes. That's
> maybe already where I have no clue how to achieve this.
>
> Maybe I'm looking into how other data stores are replicated and
> partitioned, first ;)
>
> There's also BookKeeper for distributing a transaction-log. But I'm not
> sure if it's overhead if I don't have to persist the changes before
> applying it to the real data-files (to provide consistency -- I don't need
> persisted write-ahead logging).
>
> kind regards
>
> Johannes
> On 06.08.19 15:20, Ilya Kasnacheev wrote:
>
> Hello!
>
> You can use ignite.affinity(cacheName).mapKeyToNode(key): it returns
> ClusterNode which is primary for this key.
>
> Not sure that I understand you about the quorum.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 6 авг. 2019 г. в 15:47, Johannes Lichtenberger <
> johannes.lichtenberger@unitedplanet.com>:
>
>> Hi,
>>
>> can I somehow query on which cluster-nodes a partitioned cache stores
>> values for specific keys? I might want to use a cache for replicating an
>> in-memory transaction log (which does not have to be persisted) to
>> replicate a document of a NoSQL document store to a few nodes.
>>
>> Thus, instead of a local cache I'd simply write into an ignite cache and
>> then would like to query which cluster-nodes have stored a specific key
>> (for instance the document name). So for instance I could tell a load
>> balancer for reads to read documents from one of the backup replicas.
>>
>> During a transaction commit I would also need this information to know
>> where to send an event to... to commit the transaction. And somehow I'd
>> love to wait for a quorum of nodes if the transaction really has
>> committed or it needs to roll back.
>>
>> kind regards
>>
>> Johannes
>>
>>
Re: Ignite Partitioned Cache / Use for an in-memory transaction-log
Posted by Johannes Lichtenberger <jo...@unitedplanet.com>.
Hi,
if I want to replicate data from one node to N-nodes (that is only a
subset of nodes in the whole cluster), could I also specify that not
only the primary is updated in synchronous mode, but some of the backup
nodes, too?
I also would like to somehow guarantee read-your-own-writes semantics.
My idea is for replicating a storage / database system to have a single
writer through which all writes are propagated.
I'd probably use something like oldest node is the master (which can be
done using Apache Ignite).
Then let's say someone decides to create a database and a first
resource. I'd have to send an event round robin to create the resource
on let's say three of 10 cluster nodes. Maybe I could use a distributed
semaphore which just accepts 3-times acquiring the semaphore (name is
database + resource concatenated). Once the initial bootstrapping of
creating directories and files happened on all 3 nodes a
write-transaction can be opened, which writes into a distributed cache.
However as I'm thinking about it the cache for this resource must be
shared on only the same 3 cluster-nodes. That's maybe already where I
have no clue how to achieve this.
Maybe I'm looking into how other data stores are replicated and
partitioned, first ;)
There's also BookKeeper for distributing a transaction-log. But I'm not
sure if it's overhead if I don't have to persist the changes before
applying it to the real data-files (to provide consistency -- I don't
need persisted write-ahead logging).
kind regards
Johannes
On 06.08.19 15:20, Ilya Kasnacheev wrote:
> Hello!
>
> You can use ignite.affinity(cacheName).mapKeyToNode(key): it returns
> ClusterNode which is primary for this key.
>
> Not sure that I understand you about the quorum.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 6 авг. 2019 г. в 15:47, Johannes Lichtenberger
> <johannes.lichtenberger@unitedplanet.com
> <ma...@unitedplanet.com>>:
>
> Hi,
>
> can I somehow query on which cluster-nodes a partitioned cache stores
> values for specific keys? I might want to use a cache for
> replicating an
> in-memory transaction log (which does not have to be persisted) to
> replicate a document of a NoSQL document store to a few nodes.
>
> Thus, instead of a local cache I'd simply write into an ignite
> cache and
> then would like to query which cluster-nodes have stored a
> specific key
> (for instance the document name). So for instance I could tell a load
> balancer for reads to read documents from one of the backup replicas.
>
> During a transaction commit I would also need this information to
> know
> where to send an event to... to commit the transaction. And
> somehow I'd
> love to wait for a quorum of nodes if the transaction really has
> committed or it needs to roll back.
>
> kind regards
>
> Johannes
>
Re: Ignite Partitioned Cache / Use for an in-memory transaction-log
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
You can use ignite.affinity(cacheName).mapKeyToNode(key): it returns
ClusterNode which is primary for this key.
Not sure that I understand you about the quorum.
Regards,
--
Ilya Kasnacheev
вт, 6 авг. 2019 г. в 15:47, Johannes Lichtenberger <
johannes.lichtenberger@unitedplanet.com>:
> Hi,
>
> can I somehow query on which cluster-nodes a partitioned cache stores
> values for specific keys? I might want to use a cache for replicating an
> in-memory transaction log (which does not have to be persisted) to
> replicate a document of a NoSQL document store to a few nodes.
>
> Thus, instead of a local cache I'd simply write into an ignite cache and
> then would like to query which cluster-nodes have stored a specific key
> (for instance the document name). So for instance I could tell a load
> balancer for reads to read documents from one of the backup replicas.
>
> During a transaction commit I would also need this information to know
> where to send an event to... to commit the transaction. And somehow I'd
> love to wait for a quorum of nodes if the transaction really has
> committed or it needs to roll back.
>
> kind regards
>
> Johannes
>
>