Re: Ignite cache partition size

[Stanislav Lukyanov <st...@gmail.com>]

It's best to have the number of partitions being a power of two, so better
to go with 32768 I think.
There are big clusters with hundreds of nodes out there, and they do use
large partition numbers sometimes - as large as 16k or 32k.

Note that it will bring some overhead on the metadata being stored, etc.
You may need to tune your heap to store the partition information.
And as Prasad has mentioned, you need to make sure you know how your system
behaves when a node fails - how much time does rebalancing takes, do your
remaining node have enough memory to restore the backups number, etc.

Stan

On Sun, Oct 27, 2019 at 7:21 PM Prasad Bhalerao <
prasadbhalerao1983@gmail.com> wrote:

> I have heard people creating 4096 but not more that.
> What's you reasoning behind creating such huge partitions.
> I have a table which contains 40+ million rows, but it has only 5 columns
> and I have kept the default partitions and getting good results out of it.
> You have 1450 columns, an interesting use case.
>
> 1) Is your app already in production?
> 2) Did you calculate the average size of single row.
> 3) How do you manage and monitor your big cluster.
> 4) Do you have any scripts or tools to stop,start, restart or rolling
> reastart the cluster or node of your cluster?
>
> I have cluster of 7 nodes, each node has a data region size of 40-50 Gb
> but whenever a segmentation happens i.e when Node goes of out cluster I
> have found it difficult to debug or investigate such issues. Most of the
> times I could not figure out the  exact reason for sengmentation and always
> ended up with blaming network issue for the segmentation.
>
> If you have already figured out all these things then well and good,
> please share your ideas to tools with usergroup if possible.
>
> If you are doing POC and if you want 24/7 availability then I would
> suggest to figure out these things as well.
>
> Thanks,
> Prasad
>
>
> On Sun 27 Oct, 2019, 6:53 AM Yong Zhao <codeboyyong@gmail.com wrote:
>
>>
>> Cool, thanks .
>> I have a quite big cluster with 100 nodes, so I guess 25600 will be good
>> partition?
>> I am using ignite as database.
>> I am creating a table with 20 million rows,each row has 1450 columns.
>> Is this a good idea?
>>
>> Thank you!
>>
>> On Sat, Oct 26, 2019 at 9:10 AM Prasad Bhalerao <
>> prasadbhalerao1983@gmail.com> wrote:
>>
>>> Please check this link...
>>>
>>>
>>> https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function
>>>
>>>
>>> Example from the ignite doc.
>>>
>>>
>>> // Preparing Apache Ignite node configuration.IgniteConfiguration cfg = new IgniteConfiguration();
>>>         // Creating a cache configuration.CacheConfiguration cacheCfg = new CacheConfiguration("myCache");
>>> // Creating the affinity function with custom setting.RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();
>>>         affFunc.setExcludeNeighbors(true);
>>>         affFunc.setPartitions(2048);
>>> // Applying the affinity function configuration.cacheCfg.setAffinity(affFunc);
>>>         // Setting the cache configuration.cfg.setCacheConfiguration(cacheCfg);
>>>
>>>
>>>
>>> On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <itdolmatov@gmail.com
>>> wrote:
>>>
>>>> Try to implement you're own affinity function. If it map any key to
>>>> numbers from 1 to n, you have n partitions
>>>>
>>>> On Sat, Oct 26, 2019, 09:46 codeboyyong <co...@gmail.com> wrote:
>>>>
>>>>> Hi can you please tell me how to config this ?
>>>>> "number of cache partitions"
>>>>>
>>>>> Thank You
>>>>> Yong
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>
>>>>