You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by amitpa <am...@nrifintech.com> on 2016/05/26 03:40:03 UTC
Affinity Key Mapping
If I have two domain objects lets say A, B, do I need to put them in the same
cache to use Affinity key mapping? Or can it work with different caches?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by Vladislav Pyatkov <vp...@gridgain.com>.
Hello,
>> From the example it doesnt appear that we can put it in different caches.
Yes, from the example in Java is not clear, but see example on Scala. This
should work well (in different caches), it clear from implementation.
>> how does Ignite figure out colocation in that case?
Ignite does this with the affinity function. Annotation @AffinitiKeyMapped
means that the evaluation of the partition (for store key-value pair) will
be on a particular field and not on the whole key.
>>how do you configure partitions for a cache?
You can set the number of partitions to the constructor affinity function
for a particular cache.
<property name="affinity">
<bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<constructor-arg value="true"/>
<constructor-arg value="1024"/>
</bean>
</property>
On Fri, May 27, 2016 at 6:33 AM, amitpa <am...@nrifintech.com> wrote:
> Hey Thanks for the reply.
>
> I am copying the example here :-
>
> / Instantiate person keys with the same company ID which is used as
> affinity
> key.
> Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
> Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");
>
> Person p1 = new Person(personKey1, ...);
> Person p2 = new Person(personKey2, ...);
>
> // Both, the company and the person objects will be cached on the same
> node.
> cache.put("myCompanyId", new Company(...));
> cache.put(personKey1, p1);
> cache.put(personKey2, p2);
>
> From the example it doesnt appear that we can put it in different caches.
> See both are putting to the same cache.
>
> Is there any exampel of using different caches for storing related domain
> objects, how does Ignite figure out colocation in that case? Sorry for the
> questions, but it got me confused.
>
> Also how do you configure partitions for a cache?
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5265.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Affinity Key Mapping
Posted by Alexei Scherbakov <al...@gmail.com>.
I have another idea.
Try to define country cache as atomic.
It shouldn't block on read when.
Of course it can't be used with transactional semantics.
2016-05-30 18:50 GMT+03:00 amitpa <am...@nrifintech.com>:
> Alexei,
>
> yes I understand with Ignite Optmistic means lock acquisition on write for
> READCOMMITED.
>
> With PESSIMISTIC it would be on read.
>
> However consider a cache like Country (contrived example )...many people
> may
> live in USA. But if someone tries to do a transaction for a entry on write
> all will lock on the same entry.
>
> This is a contrived example, and I can ofcourse find multiple ways to
> circumvent this particular scenario.
>
> But for some cases this can not be done.
>
> And if there is any cache config which says a cache is read only that
> should
> be fast enough.
>
> However I wil try your work around.
>
> Many thanks for your replies, it is really helpful.
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5327.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Alexei Scherbakov
Re: Affinity Key Mapping
Posted by amitpa <am...@nrifintech.com>.
Thanks everyone.
I got around this with Ignite Atomic cache.
However with replication my transactions still seem to be too slow.
Is there anything I can do to make it faster?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5360.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by Denis Magda <dm...@gridgain.com>.
Hi,
There is already a solution for your use case - you can use IgniteCache.getAllOutTx(…). No locks will be acquired for the keys passed to this method.
Alternatively, as it’s already suggested, you can make such caches ATOMIC if you don’t consider to use them in transactional mode.
—
Denis
> On May 30, 2016, at 6:50 PM, amitpa <am...@nrifintech.com> wrote:
>
> Alexei,
>
> yes I understand with Ignite Optmistic means lock acquisition on write for
> READCOMMITED.
>
> With PESSIMISTIC it would be on read.
>
> However consider a cache like Country (contrived example )...many people may
> live in USA. But if someone tries to do a transaction for a entry on write
> all will lock on the same entry.
>
> This is a contrived example, and I can ofcourse find multiple ways to
> circumvent this particular scenario.
>
> But for some cases this can not be done.
>
> And if there is any cache config which says a cache is read only that should
> be fast enough.
>
> However I wil try your work around.
>
> Many thanks for your replies, it is really helpful.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5327.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by amitpa <am...@nrifintech.com>.
Alexei,
yes I understand with Ignite Optmistic means lock acquisition on write for
READCOMMITED.
With PESSIMISTIC it would be on read.
However consider a cache like Country (contrived example )...many people may
live in USA. But if someone tries to do a transaction for a entry on write
all will lock on the same entry.
This is a contrived example, and I can ofcourse find multiple ways to
circumvent this particular scenario.
But for some cases this can not be done.
And if there is any cache config which says a cache is read only that should
be fast enough.
However I wil try your work around.
Many thanks for your replies, it is really helpful.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5327.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by Alexei Scherbakov <al...@gmail.com>.
Ignite's lock acquisition depends on the transaction isolation.
AFAIK, read_commited and optimistic modes do not lock entries on read.
Ignte has no meaning of read-only cache and I doubt it will have in future,
but you can try to ask on the dev list.
As a workaround, you can create local ConcurrentHashMap and populate it
with static data using the broadcast job [1]
on cluster startup.
[1] See javadoc for ignite.compute().broadcast()
2016-05-30 17:52 GMT+03:00 amitpa <am...@nrifintech.com>:
> Ignore my latest question . This seems to work fine.
>
> I have one kind of feature request though.
>
> It seems with transaction, Ignite always tries to get the lock for the keys
> accessed.
> However some caches can be read only. Multiple transactions can access the
> same key, but they will never modify it.
>
> Is there any way to mark a cache as read only and hence exempt for
> transaction lock acquiring, this I believe is a common case for things like
> reference data. This will also help in throughput for most applications
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5321.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Alexei Scherbakov
Re: Affinity Key Mapping
Posted by amitpa <am...@nrifintech.com>.
Ignore my latest question . This seems to work fine.
I have one kind of feature request though.
It seems with transaction, Ignite always tries to get the lock for the keys
accessed.
However some caches can be read only. Multiple transactions can access the
same key, but they will never modify it.
Is there any way to mark a cache as read only and hence exempt for
transaction lock acquiring, this I believe is a common case for things like
reference data. This will also help in throughput for most applications
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5321.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by amitpa <am...@nrifintech.com>.
Val,
Excellent feedback. I will update and let you guys know. However is there
any way to use transactions within
https://apacheignite.readme.io/docs/affinity-collocation#collocating-compute-with-data
...
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5318.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by vkulichenko <va...@gmail.com>.
I responded regarding the performance in this thread:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Transaction-performance-td5085.html
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5316.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by amitpa <am...@nrifintech.com>.
disruptor.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/n5293/disruptor.zip>
Uploading the code.
I have tried to mimic my code as much as possible. However I have used no
DTO's (only one). In my actual business code there are lots of caches with
Key Long and Value DTO. I understand the performance will be slower in that
code due to that. But that is extremely slow though too.
The main code is in Disruptor server.
IgniteMain starts another node.
The App is basically client code.
Once I run two nodes I am getting a TPS of 1100.
I have used OFF_HEAP tiered and ONHEAP tiered. The offheap seems to give
more stable performance.
However I would like to improve the performance to at least 3000 TPS. I get
over 10000 TPS on a single node (when I run only Disruptor server).
My config is i7 6700 with 32 GB ram. When Ignite boots up both nodes it
shows 14 gb memory.
Can some one provide some performance tips on this?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5293.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by Alexei Scherbakov <al...@gmail.com>.
1. Increasing number of partitions might improve throughput of Ignite
cluster, because more partitions means less contention on cache locks.
It's application dependant.
2. Affinity mapping will improve performance if puts are invoked from
affinity node ( using affinityCall [1] ), because puts need not to be
transferred over wire ( except for backups ).
3. If you have unexpected performance problems with Ignite, you can share
you code and someone from community may give you a hint.
[1]
https://apacheignite.readme.io/docs/affinity-collocation#collocating-compute-with-data
2016-05-28 12:03 GMT+03:00 amitpa <am...@nrifintech.com>:
> Does Affinity Key imporve performance of multiple cache puts in the same
> transaction?
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5290.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Alexei Scherbakov
Re: Affinity Key Mapping
Posted by amitpa <am...@nrifintech.com>.
Does Affinity Key imporve performance of multiple cache puts in the same
transaction?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5290.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by Vladislav Pyatkov <vp...@gridgain.com>.
Number of partition does not affect the performance of the transaction.
In general changing the number of partitions is done to evenly distribute
data over nodes.
On Fri, May 27, 2016 at 12:06 PM, amitpa <am...@nrifintech.com> wrote:
> Hey , many thanks for the quick and awesome response. I have an idea now
> about how to use Affinity mapping.
>
> The doc is clear now. I will ask some questions later on about Affinity
> mapping.
>
> But does increasing partitions for a cache help with Transaction
> performance? I am struggling to increase increase transaction performance
> under my usecase (due to some mis understanding on my part may be).
>
> I have already submitted a thread, where I will submit the code. However it
> will still be helpful if I can get some tips? I wanted to configure
> Affinity
> mapping for throughput of transactions. Any other tips are welcome
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5272.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Affinity Key Mapping
Posted by amitpa <am...@nrifintech.com>.
Hey , many thanks for the quick and awesome response. I have an idea now
about how to use Affinity mapping.
The doc is clear now. I will ask some questions later on about Affinity
mapping.
But does increasing partitions for a cache help with Transaction
performance? I am struggling to increase increase transaction performance
under my usecase (due to some mis understanding on my part may be).
I have already submitted a thread, where I will submit the code. However it
will still be helpful if I can get some tips? I wanted to configure Affinity
mapping for throughput of transactions. Any other tips are welcome
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5272.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by Denis Magda <dm...@gridgain.com>.
I’ve fixed the examples in the docs [1]. Thanks pointing out to this!
Also you may refer to CacheQueryExample.initialized() method to see how it works in practice. The example is delivered as a part of Ignite release bundles. [2]
Feel free to ask if anythings is still unclear.
[1] https://apacheignite.readme.io/docs/affinity-collocation <https://apacheignite.readme.io/docs/affinity-collocation>
[2] https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheQueryExample.java <https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheQueryExample.java>
—
Denis
> On May 27, 2016, at 6:33 AM, amitpa <am...@nrifintech.com> wrote:
>
> Hey Thanks for the reply.
>
> I am copying the example here :-
>
> / Instantiate person keys with the same company ID which is used as affinity
> key.
> Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
> Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");
>
> Person p1 = new Person(personKey1, ...);
> Person p2 = new Person(personKey2, ...);
>
> // Both, the company and the person objects will be cached on the same node.
> cache.put("myCompanyId", new Company(...));
> cache.put(personKey1, p1);
> cache.put(personKey2, p2);
>
> From the example it doesnt appear that we can put it in different caches.
> See both are putting to the same cache.
>
> Is there any exampel of using different caches for storing related domain
> objects, how does Ignite figure out colocation in that case? Sorry for the
> questions, but it got me confused.
>
> Also how do you configure partitions for a cache?
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5265.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by amitpa <am...@nrifintech.com>.
Hey Thanks for the reply.
I am copying the example here :-
/ Instantiate person keys with the same company ID which is used as affinity
key.
Object personKey1 = new PersonKey("myPersonId1", "myCompanyId");
Object personKey2 = new PersonKey("myPersonId2", "myCompanyId");
Person p1 = new Person(personKey1, ...);
Person p2 = new Person(personKey2, ...);
// Both, the company and the person objects will be cached on the same node.
cache.put("myCompanyId", new Company(...));
cache.put(personKey1, p1);
cache.put(personKey2, p2);
From the example it doesnt appear that we can put it in different caches.
See both are putting to the same cache.
Is there any exampel of using different caches for storing related domain
objects, how does Ignite figure out colocation in that case? Sorry for the
questions, but it got me confused.
Also how do you configure partitions for a cache?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212p5265.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Affinity Key Mapping
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hello,
You can put objects in a single cache and use affinity collocation.
But I would recommend using a different caches, but do not forget, they
should be equally configured (same number of partitions and same affinity
function).
See also https://apacheignite.readme.io/docs/affinity-collocation
On Thu, May 26, 2016 at 6:40 AM, amitpa <am...@nrifintech.com> wrote:
> If I have two domain objects lets say A, B, do I need to put them in the
> same
> cache to use Affinity key mapping? Or can it work with different caches?
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Affinity-Key-Mapping-tp5212.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov