You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by colinc <co...@yahoo.co.uk> on 2017/11/27 10:38:14 UTC
Ignite cache partition size
Normally when storing data in Ignite using the default
RendezVousAffinityFunction, data is distributed reasonably evenly over the
available nodes. When increasing the cluster size (up to 24 nodes in my
case), I'm finding that the data is not so well distributed. Some nodes have
more than twice as much data as others.
The solution I found is to quadruple the number of cache partitions up to
8192. This reduces the spread significantly - to about 25%.
Is there any down-side to configuring a large number of partitions for a
cache? Specifically:
* Am I right in observing that is slows down partition map exchange? Is this
likely to cause any problems?
* When using cross-cache data affinity, do all participating caches need to
have the same number of partitions configured for this to work?
Is there any rule of thumb for choosing the best configuration? 1000 times
the number of nodes, perhaps?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite cache partition size
Posted by Stanislav Lukyanov <st...@gmail.com>.
It's best to have the number of partitions being a power of two, so better
to go with 32768 I think.
There are big clusters with hundreds of nodes out there, and they do use
large partition numbers sometimes - as large as 16k or 32k.
Note that it will bring some overhead on the metadata being stored, etc.
You may need to tune your heap to store the partition information.
And as Prasad has mentioned, you need to make sure you know how your system
behaves when a node fails - how much time does rebalancing takes, do your
remaining node have enough memory to restore the backups number, etc.
Stan
On Sun, Oct 27, 2019 at 7:21 PM Prasad Bhalerao <
prasadbhalerao1983@gmail.com> wrote:
> I have heard people creating 4096 but not more that.
> What's you reasoning behind creating such huge partitions.
> I have a table which contains 40+ million rows, but it has only 5 columns
> and I have kept the default partitions and getting good results out of it.
> You have 1450 columns, an interesting use case.
>
> 1) Is your app already in production?
> 2) Did you calculate the average size of single row.
> 3) How do you manage and monitor your big cluster.
> 4) Do you have any scripts or tools to stop,start, restart or rolling
> reastart the cluster or node of your cluster?
>
> I have cluster of 7 nodes, each node has a data region size of 40-50 Gb
> but whenever a segmentation happens i.e when Node goes of out cluster I
> have found it difficult to debug or investigate such issues. Most of the
> times I could not figure out the exact reason for sengmentation and always
> ended up with blaming network issue for the segmentation.
>
> If you have already figured out all these things then well and good,
> please share your ideas to tools with usergroup if possible.
>
> If you are doing POC and if you want 24/7 availability then I would
> suggest to figure out these things as well.
>
> Thanks,
> Prasad
>
>
> On Sun 27 Oct, 2019, 6:53 AM Yong Zhao <codeboyyong@gmail.com wrote:
>
>>
>> Cool, thanks .
>> I have a quite big cluster with 100 nodes, so I guess 25600 will be good
>> partition?
>> I am using ignite as database.
>> I am creating a table with 20 million rows,each row has 1450 columns.
>> Is this a good idea?
>>
>> Thank you!
>>
>> On Sat, Oct 26, 2019 at 9:10 AM Prasad Bhalerao <
>> prasadbhalerao1983@gmail.com> wrote:
>>
>>> Please check this link...
>>>
>>>
>>> https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function
>>>
>>>
>>> Example from the ignite doc.
>>>
>>>
>>> // Preparing Apache Ignite node configuration.IgniteConfiguration cfg = new IgniteConfiguration();
>>> // Creating a cache configuration.CacheConfiguration cacheCfg = new CacheConfiguration("myCache");
>>> // Creating the affinity function with custom setting.RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();
>>> affFunc.setExcludeNeighbors(true);
>>> affFunc.setPartitions(2048);
>>> // Applying the affinity function configuration.cacheCfg.setAffinity(affFunc);
>>> // Setting the cache configuration.cfg.setCacheConfiguration(cacheCfg);
>>>
>>>
>>>
>>> On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <itdolmatov@gmail.com
>>> wrote:
>>>
>>>> Try to implement you're own affinity function. If it map any key to
>>>> numbers from 1 to n, you have n partitions
>>>>
>>>> On Sat, Oct 26, 2019, 09:46 codeboyyong <co...@gmail.com> wrote:
>>>>
>>>>> Hi can you please tell me how to config this ?
>>>>> "number of cache partitions"
>>>>>
>>>>> Thank You
>>>>> Yong
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>
>>>>
Re: Ignite cache partition size
Posted by Prasad Bhalerao <pr...@gmail.com>.
I have heard people creating 4096 but not more that.
What's you reasoning behind creating such huge partitions.
I have a table which contains 40+ million rows, but it has only 5 columns
and I have kept the default partitions and getting good results out of it.
You have 1450 columns, an interesting use case.
1) Is your app already in production?
2) Did you calculate the average size of single row.
3) How do you manage and monitor your big cluster.
4) Do you have any scripts or tools to stop,start, restart or rolling
reastart the cluster or node of your cluster?
I have cluster of 7 nodes, each node has a data region size of 40-50 Gb but
whenever a segmentation happens i.e when Node goes of out cluster I have
found it difficult to debug or investigate such issues. Most of the times I
could not figure out the exact reason for sengmentation and always ended
up with blaming network issue for the segmentation.
If you have already figured out all these things then well and good, please
share your ideas to tools with usergroup if possible.
If you are doing POC and if you want 24/7 availability then I would suggest
to figure out these things as well.
Thanks,
Prasad
On Sun 27 Oct, 2019, 6:53 AM Yong Zhao <codeboyyong@gmail.com wrote:
>
> Cool, thanks .
> I have a quite big cluster with 100 nodes, so I guess 25600 will be good
> partition?
> I am using ignite as database.
> I am creating a table with 20 million rows,each row has 1450 columns.
> Is this a good idea?
>
> Thank you!
>
> On Sat, Oct 26, 2019 at 9:10 AM Prasad Bhalerao <
> prasadbhalerao1983@gmail.com> wrote:
>
>> Please check this link...
>>
>>
>> https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function
>>
>>
>> Example from the ignite doc.
>>
>>
>> // Preparing Apache Ignite node configuration.IgniteConfiguration cfg = new IgniteConfiguration();
>> // Creating a cache configuration.CacheConfiguration cacheCfg = new CacheConfiguration("myCache");
>> // Creating the affinity function with custom setting.RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();
>> affFunc.setExcludeNeighbors(true);
>> affFunc.setPartitions(2048);
>> // Applying the affinity function configuration.cacheCfg.setAffinity(affFunc);
>> // Setting the cache configuration.cfg.setCacheConfiguration(cacheCfg);
>>
>>
>>
>> On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <itdolmatov@gmail.com wrote:
>>
>>> Try to implement you're own affinity function. If it map any key to
>>> numbers from 1 to n, you have n partitions
>>>
>>> On Sat, Oct 26, 2019, 09:46 codeboyyong <co...@gmail.com> wrote:
>>>
>>>> Hi can you please tell me how to config this ?
>>>> "number of cache partitions"
>>>>
>>>> Thank You
>>>> Yong
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>
Re: Ignite cache partition size
Posted by Yong Zhao <co...@gmail.com>.
Cool, thanks .
I have a quite big cluster with 100 nodes, so I guess 25600 will be good
partition?
I am using ignite as database.
I am creating a table with 20 million rows,each row has 1450 columns.
Is this a good idea?
Thank you!
On Sat, Oct 26, 2019 at 9:10 AM Prasad Bhalerao <
prasadbhalerao1983@gmail.com> wrote:
> Please check this link...
>
>
> https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function
>
>
> Example from the ignite doc.
>
>
> // Preparing Apache Ignite node configuration.IgniteConfiguration cfg = new IgniteConfiguration();
> // Creating a cache configuration.CacheConfiguration cacheCfg = new CacheConfiguration("myCache");
> // Creating the affinity function with custom setting.RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();
> affFunc.setExcludeNeighbors(true);
> affFunc.setPartitions(2048);
> // Applying the affinity function configuration.cacheCfg.setAffinity(affFunc);
> // Setting the cache configuration.cfg.setCacheConfiguration(cacheCfg);
>
>
>
> On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <itdolmatov@gmail.com wrote:
>
>> Try to implement you're own affinity function. If it map any key to
>> numbers from 1 to n, you have n partitions
>>
>> On Sat, Oct 26, 2019, 09:46 codeboyyong <co...@gmail.com> wrote:
>>
>>> Hi can you please tell me how to config this ?
>>> "number of cache partitions"
>>>
>>> Thank You
>>> Yong
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
Re: Ignite cache partition size
Posted by Prasad Bhalerao <pr...@gmail.com>.
Please check this link...
https://apacheignite.readme.io/docs/affinity-collocation#section-affinity-function
Example from the ignite doc.
// Preparing Apache Ignite node configuration.IgniteConfiguration cfg
= new IgniteConfiguration();
// Creating a cache configuration.CacheConfiguration cacheCfg
= new CacheConfiguration("myCache");
// Creating the affinity function with custom
setting.RendezvousAffinityFunction affFunc = new
RendezvousAffinityFunction();
affFunc.setExcludeNeighbors(true);
affFunc.setPartitions(2048);
// Applying the affinity function configuration.cacheCfg.setAffinity(affFunc);
// Setting the cache configuration.cfg.setCacheConfiguration(cacheCfg);
On Sat 26 Oct, 2019, 7:42 PM Andrey Dolmatov <itdolmatov@gmail.com wrote:
> Try to implement you're own affinity function. If it map any key to
> numbers from 1 to n, you have n partitions
>
> On Sat, Oct 26, 2019, 09:46 codeboyyong <co...@gmail.com> wrote:
>
>> Hi can you please tell me how to config this ?
>> "number of cache partitions"
>>
>> Thank You
>> Yong
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: Ignite cache partition size
Posted by Andrey Dolmatov <it...@gmail.com>.
Try to implement you're own affinity function. If it map any key to numbers
from 1 to n, you have n partitions
On Sat, Oct 26, 2019, 09:46 codeboyyong <co...@gmail.com> wrote:
> Hi can you please tell me how to config this ?
> "number of cache partitions"
>
> Thank You
> Yong
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite cache partition size
Posted by codeboyyong <co...@gmail.com>.
Hi can you please tell me how to config this ?
"number of cache partitions"
Thank You
Yong
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite cache partition size
Posted by "ilya.kasnacheev" <il...@gmail.com>.
Hello!
* 8192 partitions have been tried with success, as far as I know. I think
you can give it a shot, see where it gets you. It might slow down partition
exchange a bit, but it should not hang just from increased amount of
information (as opposed to stuck operations)
* Yes,they need to have the same number of partitions for caches with
collocated keys. Come to think of it, if all of 2N partitions were
collocated with 1N partitions, what you will really have is just 1N
partitions. Partitions should be able to part in order to work.
For optimal number of partitions, consider 256 primary partitions per node.
This means 8192 is OK for you until you move past 32 nodes.
Some additional hints:
* Increasing number of partitions may increase number of open files if using
persistence. Plan accordingly, keep ulimits in mind.
* You can broadcast ignite.affinity(cacheName).primaryPartitions(localNode)
and ignite.cache(cacheName).localSize(CachePeekMode.PRIMARY) to data nodes
to get hold of how even partition distribution is in your case (are keys
distributed well between partitions? are partitions between nodes?
* There was a research on how well partitions are distributed between nodes
with Rendevouz affinity, see non-"balanced" PNGs (that's per # of nodes) in
https://issues.apache.org/jira/browse/IGNITE-3018 - I think you can observe
serious deviations from average number of partitions in those images, so
it's possible that your 2x difference is no accident.
* The community is currently working on faster and fairer affinity functions
:)
Regards,
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/