Overall desgn question

[Scott Tyriver <Sc...@mgic.com>]

We will be employing a data grid off heap, replication implementation. We 
are estimating with the 30% index cache alllowance that we will need 7 gig 
of ram per ignite server node there will be two. We have two  options. 
Create many smaller caches or put everything in a collection using one 
cache.

Are there any recommendations that someone might have as to the best 
approach?



This message is intended for use only by the person(s) addressed above and 
may contain privileged and confidential information. Disclosure or use of 
this message by any other person is strictly prohibited. If this message 
is received in error, please notify the sender immediately and delete this 
message.

Re: Overall desgn question

[vkulichenko <va...@gmail.com>]

With replicated cache you don't need to collocate. You can try both
approaches - a single cache for documents that have nested collection of
pages, or two separate caches for documents and pages.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Overall-desgn-question-tp7459p7613.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Overall desgn question

[styriver <Sc...@mgic.com>]

Thank you for the response. Does this approach make sense if we are going to
be in replicated mode. We will not be using partition mode. 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Overall-desgn-question-tp7459p7611.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Overall desgn question

[vkulichenko <va...@gmail.com>]

Cache per document doesn't make much sense. I was talking about cache per
data type.

Ideally, you should split Document and Page objects (instead of creating a
nested map within the document), store each page as a separate document and
use collocation [1] to make sure that all pages are stored next to
corresponding documents. In this case you will end up with two caches - one
for documents and another for pages. You can then join them using SQL
queries [2].

[1]
https://apacheignite.readme.io/docs/affinity-collocation#collocate-data-with-data
[2] https://apacheignite.readme.io/docs/sql-queries#cross-cache-queries

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Overall-desgn-question-tp7459p7495.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Overall desgn question

[styriver <Sc...@mgic.com>]

One performance question we had would be serialization. If we do a cache per
document strategy we were thinking the serialization cost would be less
rather than having to go thru serialization of the entire nested HashMap.

Again we are going to use Off Heap in replicated mode.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Overall-desgn-question-tp7459p7490.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Overall desgn question

[styriver <Sc...@mgic.com>]

Thanks for the response. At our peak we could have around 1000 documents
cached. We can go with a cache per document design or a design with one
named cache and add the document to the cache along with a nested hash map
representing the pages.

IgniteCache<String(doc Id), HashMap&lt;String (Page#), ImagePage>>

Are there any performance considerations between a cache per document or one
cache with a nested HashMap? Based on your previous response at peak it
looks like we could be adding 2 gig of heap space by going with a Cache Per
Document design. So memory utilization would be lower with the one named
cache approach just was wondering if there are any performance
considerations between the two approaches



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Overall-desgn-question-tp7459p7488.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Overall desgn question

[vkulichenko <va...@gmail.com>]

Hi Scott,

How many caches are you going to have? Basically, the recommended way is to
have a cache per data type - this makes life easier in most cases. But keep
in mind that each cache will require around 20MB of *heap* memory per node
for its internals. So hundreds of caches can introduce high memory
consumption and you may consider merging some caches in this case.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Overall-desgn-question-tp7459p7466.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.