[jira] [Comment Edited] (IGNITE-4154) Optimize amount of data stored in discovery history

["Semen Boikov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656992#comment-15656992 ] 

Semen Boikov edited comment on IGNITE-4154 at 11/11/16 12:48 PM:
-----------------------------------------------------------------

Done these optimizations as part of this issue:

Discovery history optimizations:
- remove discarded message for discovery pending messages
- remove duplicated data from TcpDiscoveryNodeAddedMessage.oldNodesDiscoData
- do not store unnecessary data in discovery EnsuredMessageHistory
- use special property for EnsuredMessageHistory size instead of IGNITE_DISCOVERY_HISTORY_SIZE

Affinity history optimizations:
- do not store calculated primary/backup maps in history
- try save the same assignments instance for caches with similar affinity

Exchange messages optimizations:
- do not send duplicated partition state maps for caches with similar affinity
- use zip compression for data sent in exchange messages



was (Author: sboikov):
Done these optimizations as part of this issue:

Discovery history optimizations:
- remove discarded message for discovery pending messages
- remove duplicated data from TcpDiscoveryNodeAddedMessage.oldNodesDiscoData
- do not store unnecessary data in discovery EnsuredMessageHistory
- use special property for EnsuredMessageHistory size instead of IGNITE_DISCOVERY_HISTORY_SIZE
Affinity history optimizations:
- do not store calculated primary/backup maps in history
- try save the same assignments instance for caches with similar affinity
Exchange messages optimizations:
- do not send duplicated partition state maps for caches with similar affinity
- use zip compression for data sent in exchange messages


> Optimize amount of data stored in discovery history
> ---------------------------------------------------
>
>                 Key: IGNITE-4154
>                 URL: https://issues.apache.org/jira/browse/IGNITE-4154
>             Project: Ignite
>          Issue Type: Task
>          Components: general
>            Reporter: Semen Boikov
>            Assignee: Semen Boikov
>             Fix For: 1.8
>
>
> Currently on large clusters with lots of caches amount of data stored in tcp discovery history can grow too much.
> What can be improved:
> - TcpDiscoveryNodeAddedMessage.oldNodesDiscoData contains per-node discovery data. For cache component essentially this is list of configured caches, and for joined node at the time when 'collectDiscoveryData' is called this list should be the same on all nodes, so it can be stored once instead of per-node map
> - even after message is discarded it is still stored in PendingMessages collection, looks like it can be safely removed after discarding
> - now system property IGNITE_DISCOVERY_HISTORY_SIZE controls both size of events needed for client reconnects and number of topology snapshots stored in GridDiscoveryManager, need have two properies
> - for clients TcpDiscoveryNodeAddFinishedMessage.clientDiscoData can be initialized only when message is sent to joining client 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)