You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Denis Magda <dm...@apache.org> on 2017/08/24 19:32:54 UTC
[DISCUSSION] Urgent Ignite bug fix release
Igniters,
Presently every Apache Ignite node requests 80% of RAM from an operating system. We debated a lot about this default [1] and as a practice shows those who voted for the lower value (20%-25%) were right.
Our users started reporting about bizarre issues ([2], [3], [4]) when a cluster hangs, freezes for a while or even crashes during standard preloading phase or minor workloads.
It’s turned out to be that lack of RAM causes extensive swapping and checkpointing activity. Even more, the issue can be reproduced on your laptop if you launch a couple of nodes and start data preloading.
All this can be avoided if you allocated less than 80% and tune the checkpointing if the persistence is used. Unfortunately, only few could get to this point, the rest would kick Ignite out from their deployments.
Thus, we have to fix the defaults ([5] and [6]) on our own and release an emergency release with the fixes as soon as possible.
Any objections or other thoughts?
[1] http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-5717-improvements-of-MemoryPolicy-default-size-td20264.html
[2] http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16337.html
[3] http://apache-ignite-users.70518.x6.nabble.com/Strange-problems-with-Ignite-native-Persistence-when-Data-exceeds-Memory-td16187.html#a16194
[4] http://apache-ignite-users.70518.x6.nabble.com/Activation-slow-and-Ignite-node-crashed-in-the-middle-of-checkpoint-td16144.html
[5] https://issues.apache.org/jira/browse/IGNITE-6003
[6] https://issues.apache.org/jira/browse/IGNITE-6182
—
Denis
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Ivan Rakov <iv...@gmail.com>.
Backport issue: https://issues.apache.org/jira/browse/IGNITE-6204
Best Regards,
Ivan Rakov
On 28.08.2017 17:19, Anton Vinogradov wrote:
> Igniters,
>
> Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> In this case all other issues with fixVersion = 2.2 should be moved to 2.3.
>
> Currently, I see 835 issues with fixVersion = 2.2
>
> Seems we should have only 4 issues with fixVersion = 2.2:
> - Change default max memory size from 80% to 20% -
> https://issues.apache.org/jira/browse/IGNITE-6182
> - Detecting low memory on ignite node startup -
> https://issues.apache.org/jira/browse/IGNITE-6003
> - Backport of improvements to checkpoint algorithm -
> https://issues.apache.org/jira/browse/IGNITE-????
> - ML profile is missing in 2.1 binary release -
> https://issues.apache.org/jira/browse/IGNITE-6193
>
> Please correct me in case I've missed something.
>
> Ivan,
>> Let's include to the release tickets with optimizations of checkpointing
> algorithm:
>> https://issues.apache.org/jira/browse/IGNITE-6178
>> https://issues.apache.org/jira/browse/IGNITE-6033
>> https://issues.apache.org/jira/browse/IGNITE-5961
>> This will help users who are experiencing problems with slow checkpoints.
>> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to
> soften "Ignite node crashed in the middle of checkpoint" message as per
> discussion <
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>> .
> All of these issues should have fixVersion 2.3.
> Please create backport issue and link it to all necessary issues.
>
> P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
>
> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <av...@gridgain.com>
> wrote:
>
>> Denis,
>>
>>> BTW, who is considered to be the release manager of this release?
>> I'll do it.
>>
>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gv...@gmail.com>
>> wrote:
>>
>>> Ok, the check happens at the node start time or on NODE_JOIN event
>>>
>>> in general it looks like:
>>>
>>> 1) calculate expected used memory = heap max + system cache max + all
>>> custom policies max + default policy size and put it into a node attribute
>>>
>>> 2) get total physycal memory, calculate expected safe to be used memory
>>> amount (leave 4 gb min or 20% of available memory for OS)
>>>
>>> 3) if expected used memory + expected used memory of other nodes on the
>>> host > than safe to be used memory amount, start calculating suggestions
>>>
>>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
>>> 100mb default polycy, if available memory is less we cannot suggest
>>> anything reasonable, print warning, stop calculation.
>>>
>>> 5) check heap size (shouldn't exceed 30% of available memory (total_memory
>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
>>> calculated value or 512MB minimal)
>>>
>>> 6) check if system cache size changed, suggest default value if it's so
>>>
>>> 7) in case 100 mb * policies count < available memory, suggest using
>>> default policy with max size equals to remaining memory (available - heap
>>> -
>>> system cache)
>>>
>>> 8) calculate new size for each memory policy ( it's user defined size *
>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>> remaining memory, devided by nodes number on the host or 100 mb minimal)
>>>
>>> 9) print suggestions
>>>
>>>
>>>
>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>
>>>> Igor, can you please describe the algorithm with all the thresholds?
>>>>
>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gvvinblade@gmail.com
>>>>
>>>> wrote:
>>>>
>>>>> The suggestion here is based on initial settings, and it's so because
>>>> there
>>>>> is no other nodes on the host in the example.
>>>>>
>>>>> The algorithm tries to preserve the original ratio of memory policies
>>>>> keeping numbers reasonable (for example after some thresshold it will
>>>>> suggest not to use several memory policies if there is not enough of
>>>> memory
>>>>> for all of them) and taking into consideration nodes count on the
>>> host,
>>>>> each jvm heap, needed memory for OS, etc
>>>>>
>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>>>
>>>>>> Looks good, but why in the example provided are we suggesting 8GB? 2
>>>>> nodes
>>>>>> with 8GB will completely exhaust the available memory. I would
>>> suggest
>>>> 6
>>>>> or
>>>>>> 7GB.
>>>>>>
>>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
>>> small.
>>>>>> Can you please comment?
>>>>>>
>>>>>> D.
>>>>>>
>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>>>> gvvinblade@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> One more example of possible warning:
>>>>>>>
>>>>>>> -----------------------------------------------------
>>>>>>> Excessive memory usage by Ignite node process (performance may
>>> drop)
>>>>>>> [requested=44613MB, available=15942MB].
>>>>>>>
>>>>>>> Please tune the folowing settings as suggested:
>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>>>>>>>
>>>>>>> Current settings:
>>>>>>> Java Heap maxSize: 3543MB
>>>>>>> Java Heap initSize: 250MB
>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>>>>>>> The overall expected memory usage by all Ignite nodes on the
>>> host:
>>>>>>> 44613MB
>>>>>>> -----------------------------------------------------
>>>>>>>
>>>>>>> Your thoughts?
>>>>>>>
>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>>>>>>>
>>>>>>>> Guys,
>>>>>>>>
>>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
>>>>> rolled
>>>>>>>> out in this emergency release:
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>>>>>>>>
>>>>>>>> Oleg, Yuri, please step in and handle the issue.
>>>>>>>>
>>>>>>>> BTW, who is considered to be the release manager of this
>>> release?
>>>>>>>> —
>>>>>>>> Denis
>>>>>>>>
>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org>
>>>>>>>> wrote:
>>>>>>>>> I like the format proposed by Denis, very clear.
>>>>>>>>>
>>>>>>>>> However, I also do not understand why a user should change the
>>>> size
>>>>>> of
>>>>>>>> some
>>>>>>>>> system cache. How would a user ever know what value to put
>>> there?
>>>>>> This
>>>>>>>>> value should be configured by Ignite automatically.
>>>>>>>>>
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>>> dmagda@apache.org>
>>>>>>> wrote:
>>>>>>>>>> Igor,
>>>>>>>>>>
>>>>>>>>>> Let me suggest this format.
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------
>>>>>>>>>> Excessive memory usage by Ignite node process (performance
>>> may
>>>>> drop)
>>>>>>>>>> [requested=29251MB, available=15942MB]
>>>>>>>>>>
>>>>>>>>>> Please tune the following settings:
>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
>>>> value]
>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
>>>>> suggested
>>>>>>>>>> value]
>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
>>>>> suggested
>>>>>>>>>> value]
>>>>>>>>>>
>>>>>>>>>> Current settings:
>>>>>>>>>> [DefaultMemoryPolicySize = value]
>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>> SystemCacheInitialSize = value
>>>>>>>>>> SystemCacheMaxSize = value
>>>>>>>>>> Java Heap Init Size = value
>>>>>>>>>> Java Heap Max Size = value
>>>>>>>>>>
>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>>> value
>>>>>>>>>> -------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Records in […] are optional. If custom memory policy is not
>>> set
>>>> or
>>>>>> the
>>>>>>>>>> default memory policy is overridden the output will miss
>>> some of
>>>>> the
>>>>>>>> rows.
>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>>>> parameter
>>>>>> was
>>>>>>>> set
>>>>>>>>>> explicitly by user code. Otherwise, the platform should be
>>> wise
>>>>>> enough
>>>>>>>> to
>>>>>>>>>> instantiate it properly depending on the host memory usage.
>>>>>>>>>>
>>>>>>>>>> —
>>>>>>>>>> Denis
>>>>>>>>>>
>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>>>>>> gvvinblade@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> The message without logging layout:
>>>>>>>>>>>
>>>>>>>>>>> Not enough memory for current process [required=29251MB,
>>>>>>>>>> available=15942MB].
>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>>> memory
>>>>>>>> allocated
>>>>>>>>>>> for each node.
>>>>>>>>>>>
>>>>>>>>>>> Current settings:
>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>
>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>>>>>>>>>>>
>>>>>>>>>>> I think it's make sense to describe what these numbers are
>>>>> consist
>>>>>>> of.
>>>>>>>>>>> We simple say which parameters have an impact on how much
>>>> memory
>>>>>> the
>>>>>>>>>>> instance needs and their (parameters) actual values.
>>>>>>>>>>>
>>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
>>>> the
>>>>>>> server
>>>>>>>>>> or
>>>>>>>>>>> workstation and it also consumes memory.
>>>>>>>>>>>
>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>>>>
>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
>>>>> text?
>>>>>>> Are
>>>>>>>> we
>>>>>>>>>>>> really going to print this whole thing out?
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> This message appears on topology change in case the
>>> available
>>>>>>> memory
>>>>>>>> is
>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>>>>>> gvvinblade@gmail.com
>>>>>>>> :
>>>>>>>>>>>>>> An example of current impl:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
>>>>> memory
>>>>>>> for
>>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>>> change
>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
>>>>> MemoryConfiguration.
>>>>>>>>>>>>> defaultMemoryPolicySize
>>>>>>>>>>>>>> to decrease memory allocated for each node.
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
>>>>> settings:
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>> HeapInit=250MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>> HeapMax=3543MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>> DefaultMemoryPolicySize=
>>>>>>>>>>>>>> 12753MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>>> ignite
>>>>>>>> instanses
>>>>>>>>>>>> on
>>>>>>>>>>>>>> the server require: 12853MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>>>>> skozlov@gridgain.com
>>>>>>> :
>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
>>> suggest to
>>>>>>> reduce
>>>>>>>>>>>> both
>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
>>>>>> available
>>>>>>>> for
>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
>>>> memory
>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> each node.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
>>> for
>>>>> each
>>>>>>>> node
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> be? In that case we should suggest it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
>>> MB.
>>>>> Use
>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
>>>>> change
>>>>>>> the
>>>>>>>>>>>>>>>> setting.
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>> memory
>>>>>>>>>>>> available
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
>>>> usage.
>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
>>> MemoryPolicyConfiguration
>>>> to
>>>>>>>>>>>> change
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> settings.
>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>>> available=15942MB]
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>>> production
>>>> for
>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> deployment consistency reasons)
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>>>>>>>>>> ['ignite-sys-cache']]
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>>> te/incubator-ignite/libs/
>>>>>>> licenses
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>>>>>> yzhdanov@apache.org
>>>>>>>> :
>>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
>>>>> mentioned
>>>>>>> by
>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> Ivan.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --Yakov
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Sergey Kozlov
>>>>>>>>>>>>>>> GridGain Systems
>>>>>>>>>>>>>>> www.gridgain.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Oleg Ostanin <oo...@gridgain.com>.
We created the release build procedure on Apache CI server according to
this ticket:
https://issues.apache.org/jira/browse/IGNITE-5249
We also changed the checksum calculation method:
https://issues.apache.org/jira/browse/IGNITE-5817
Oleg
On Thu, Sep 7, 2017 at 9:49 AM, 李玉珏@163 <18...@163.com> wrote:
> Currently,the latest version is apache-ignite-fabric-2.1.0-bin.zip <
> http://mirrors.hust.edu.cn/apache//ignite/2.1.0/apache-igni
> te-fabric-2.1.0-bin.zip>.
>
> and in http://www.gridgainsystems.com/nexus/content/repositories/
> external/org/apache/ignite/ignite-core/
>
> the latest version is 2.1.4,
>
> As I understand it, add 1 on the version number of 2.1.4, and then publish
> it to the ignite main website to ensure that both versions are
> consistent,is it ok?
>
>
> 在 2017/9/1 下午6:57, Anton Vinogradov 写道:
>
>> Hi,
>>
>> Reasonable question, but I propose to keep 2.2 as a release version for
>> this urgent release, since we never released X.Y.Z version before.
>>
>> Let's discuss this at separate thread.
>> Could you please start it?
>>
>>
>> On Fri, Sep 1, 2017 at 4:49 AM, 李玉珏@163 <18624049226@163.com <mailto:
>> 18624049226@163.com>> wrote:
>>
>> Since it's an urgent bugfix version, why can't the version number
>> be defined as 2.1.1 or 2.1.4?
>> After all, functionality has not increased and documents do not
>> require big changes.
>> If the version number is 2.1 series, there is no need to publish a
>> new version of the document.
>>
>>
>>
>> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
>>
>> Igniters,
>>
>> Seems 2.2 is a urgent bugfix release, so it should be based on
>> 2.1,
>> In this case all other issues with fixVersion = 2.2 should be
>> moved to 2.3.
>>
>> Currently, I see 835 issues with fixVersion = 2.2
>>
>> Seems we should have only 4 issues with fixVersion = 2.2:
>> - Change default max memory size from 80% to 20% -
>> https://issues.apache.org/jira/browse/IGNITE-6182
>> <https://issues.apache.org/jira/browse/IGNITE-6182>
>> - Detecting low memory on ignite node startup -
>> https://issues.apache.org/jira/browse/IGNITE-6003
>> <https://issues.apache.org/jira/browse/IGNITE-6003>
>> - Backport of improvements to checkpoint algorithm -
>> https://issues.apache.org/jira/browse/IGNITE-??
>> <https://issues.apache.org/jira/browse/IGNITE-??>??
>> - ML profile is missing in 2.1 binary release -
>> https://issues.apache.org/jira/browse/IGNITE-6193
>> <https://issues.apache.org/jira/browse/IGNITE-6193>
>>
>> Please correct me in case I've missed something.
>>
>> Ivan,
>>
>> Let's include to the release tickets with optimizations of
>> checkpointing
>>
>> algorithm:
>>
>> https://issues.apache.org/jira/browse/IGNITE-6178
>> <https://issues.apache.org/jira/browse/IGNITE-6178>
>> https://issues.apache.org/jira/browse/IGNITE-6033
>> <https://issues.apache.org/jira/browse/IGNITE-6033>
>> https://issues.apache.org/jira/browse/IGNITE-5961
>> <https://issues.apache.org/jira/browse/IGNITE-5961>
>> This will help users who are experiencing problems with
>> slow checkpoints.
>> Also, let's include
>> https://issues.apache.org/jira/browse/IGNITE-6183
>> <https://issues.apache.org/jira/browse/IGNITE-6183> to
>>
>> soften "Ignite node crashed in the middle of checkpoint"
>> message as per
>> discussion <
>> http://apache-ignite-developers.2346864.n4.nabble.com/
>> Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>> <http://apache-ignite-developers.2346864.n4.nabble.com/
>> Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html>
>>
>> .
>>
>> All of these issues should have fixVersion 2.3.
>> Please create backport issue and link it to all necessary issues.
>>
>> P.s. I've created branch ignite-2.2, currently it equals to
>> ignite-2.1.
>>
>> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov
>> <avinogradov@gridgain.com <ma...@gridgain.com>>
>> wrote:
>>
>> Denis,
>>
>> BTW, who is considered to be the release manager of
>> this release?
>>
>> I'll do it.
>>
>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor
>> <gvvinblade@gmail.com <ma...@gmail.com>>
>> wrote:
>>
>> Ok, the check happens at the node start time or on
>> NODE_JOIN event
>>
>> in general it looks like:
>>
>> 1) calculate expected used memory = heap max + system
>> cache max + all
>> custom policies max + default policy size and put it
>> into a node attribute
>>
>> 2) get total physycal memory, calculate expected safe
>> to be used memory
>> amount (leave 4 gb min or 20% of available memory for OS)
>>
>> 3) if expected used memory + expected used memory of
>> other nodes on the
>> host > than safe to be used memory amount, start
>> calculating suggestions
>>
>> 4) Each ignite instance needs at least 512mb heap +
>> 40mb system cache +
>> 100mb default polycy, if available memory is less we
>> cannot suggest
>> anything reasonable, print warning, stop calculation.
>>
>> 5) check heap size (shouldn't exceed 30% of available
>> memory (total_memory
>> - reserved for OS memory) * 30% for all JVMs, if it
>> exeedes, suggest just
>> calculated value or 512MB minimal)
>>
>> 6) check if system cache size changed, suggest default
>> value if it's so
>>
>> 7) in case 100 mb * policies count < available memory,
>> suggest using
>> default policy with max size equals to remaining
>> memory (available - heap
>> -
>> system cache)
>>
>> 8) calculate new size for each memory policy ( it's
>> user defined size *
>> (remaining / (all_policies_size * nodes_cnt)); in
>> proportion to
>> remaining memory, devided by nodes number on the host
>> or 100 mb minimal)
>>
>> 9) print suggestions
>>
>>
>>
>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan
>> <dsetrakyan@apache.org <ma...@apache.org>>:
>>
>> Igor, can you please describe the algorithm with
>> all the thresholds?
>>
>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor
>> <gvvinblade@gmail.com <ma...@gmail.com>
>>
>> wrote:
>>
>> The suggestion here is based on initial
>> settings, and it's so because
>>
>> there
>>
>> is no other nodes on the host in the example.
>>
>> The algorithm tries to preserve the original
>> ratio of memory policies
>> keeping numbers reasonable (for example after
>> some thresshold it will
>> suggest not to use several memory policies if
>> there is not enough of
>>
>> memory
>>
>> for all of them) and taking into consideration
>> nodes count on the
>>
>> host,
>>
>> each jvm heap, needed memory for OS, etc
>>
>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan
>> <dsetrakyan@apache.org
>> <ma...@apache.org>>:
>>
>> Looks good, but why in the example
>> provided are we suggesting 8GB? 2
>>
>> nodes
>>
>> with 8GB will completely exhaust the
>> available memory. I would
>>
>> suggest
>>
>> 6
>>
>> or
>>
>> 7GB.
>>
>> Also, why 100MB for default policy.
>> Anything under 1GB seems too
>>
>> small.
>>
>> Can you please comment?
>>
>> D.
>>
>> On Mon, Aug 28, 2017 at 3:31 AM,
>> Seliverstov Igor <
>>
>> gvvinblade@gmail.com <ma...@gmail.com>>
>>
>> wrote:
>>
>> One more example of possible warning:
>>
>> ------------------------------
>> -----------------------
>> Excessive memory usage by Ignite node
>> process (performance may
>>
>> drop)
>>
>> [requested=44613MB, available=15942MB].
>>
>> Please tune the folowing settings as
>> suggested:
>>
>> MemoryPolicyConfiguration.initialSize
>> for bigPlc: 8102MB
>> MemoryPolicyConfiguration.maxSize
>> for bigPlc: 8102MB
>>
>> MemoryPolicyConfiguration.initialSize
>> for dfltPlc: 100MB
>> MemoryPolicyConfiguration.maxSize
>> for dfltPlc: 100MB
>>
>> Current settings:
>> Java Heap maxSize: 3543MB
>> Java Heap initSize: 250MB
>>
>> MemoryPolicyConfiguration.initialSize
>> for bigPlc: 256MB
>> MemoryPolicyConfiguration.maxSize
>> for bigPlc: 40960MB
>>
>> MemoryPolicyConfiguration.initialSize
>> for dfltPlc: 10MB
>> MemoryPolicyConfiguration.maxSize
>> for dfltPlc: 10MB
>> The overall expected memory usage
>> by all Ignite nodes on the
>>
>> host:
>>
>> 44613MB
>> ------------------------------
>> -----------------------
>>
>> Your thoughts?
>>
>> 2017-08-28 5:06 GMT+03:00 Denis Magda
>> <dmagda@apache.org
>> <ma...@apache.org>>:
>>
>> Guys,
>>
>> ML lib profile is missing in 2.1
>> release! That must be fixed and
>>
>> rolled
>>
>> out in this emergency release:
>> https://issues.apache.org/jira
>> /browse/IGNITE-6193
>> <https://issues.apache.org/jir
>> a/browse/IGNITE-6193>
>> <
>> https://issues.apache.org/jira
>> /browse/IGNITE-6193
>> <https://issues.apache.org/jir
>> a/browse/IGNITE-6193>>
>>
>> Oleg, Yuri, please step in and
>> handle the issue.
>>
>> BTW, who is considered to be the
>> release manager of this
>>
>> release?
>>
>> —
>> Denis
>>
>> On Aug 25, 2017, at 2:29 PM,
>> Dmitriy Setrakyan <
>>
>> dsetrakyan@apache.org
>> <ma...@apache.org>>
>>
>> wrote:
>>
>> I like the format proposed by
>> Denis, very clear.
>>
>> However, I also do not
>> understand why a user should
>> change the
>>
>> size
>>
>> of
>>
>> some
>>
>> system cache. How would a user
>> ever know what value to put
>>
>> there?
>>
>> This
>>
>> value should be configured by
>> Ignite automatically.
>>
>> D.
>>
>> On Fri, Aug 25, 2017 at 2:24
>> PM, Denis Magda <
>>
>> dmagda@apache.org <ma...@apache.org>>
>>
>> wrote:
>>
>> Igor,
>>
>> Let me suggest this format.
>>
>> ------------------------------
>> ---------------
>> Excessive memory usage by
>> Ignite node process
>> (performance
>>
>> may
>>
>> drop)
>>
>> [requested=29251MB,
>> available=15942MB]
>>
>> Please tune the following
>> settings:
>>
>> [MemoryConfiguration.defaultMemoryPolicySize
>> = suggested
>>
>> value]
>>
>>
>> MemoryConfiguration.systemCacheMaxSize
>> = suggested value
>>
>> [MemoryPolicyConfiguration.maxSize
>> for {policy_name_1} =
>>
>> suggested
>>
>> value]
>>
>> [MemoryPolicyConfiguration.maxSize
>> for {policy_name_2} =
>>
>> suggested
>>
>> value]
>>
>> Current settings:
>>
>> [DefaultMemoryPolicySize
>> = value]
>> [{policy_name_1} size =
>> value]
>> [{policy_name_1} size =
>> value]
>> SystemCacheInitialSize
>> = value
>> SystemCacheMaxSize = value
>> Java Heap Init Size = value
>> Java Heap Max Size = value
>>
>> The overall memory usage
>> by all Ignite nodes on the
>> host:
>>
>> value
>>
>> ------------------------------
>> -------------
>>
>> Records in […] are
>> optional. If custom memory
>> policy is not
>>
>> set
>>
>> or
>>
>> the
>>
>> default memory policy is
>> overridden the output will
>> miss
>>
>> some of
>>
>> the
>>
>> rows.
>>
>> As for systemCacheMaxSize,
>> it should be show ONLY if the
>>
>> parameter
>>
>> was
>>
>> set
>>
>> explicitly by user code.
>> Otherwise, the platform
>> should be
>>
>> wise
>>
>> enough
>>
>> to
>>
>> instantiate it properly
>> depending on the host
>> memory usage.
>>
>> —
>> Denis
>>
>> On Aug 25, 2017, at
>> 1:49 PM, Seliverstov
>> Igor <
>>
>> gvvinblade@gmail.com
>> <ma...@gmail.com>>
>>
>> wrote:
>>
>> The message without
>> logging layout:
>>
>> Not enough memory for
>> current process
>> [required=29251MB,
>>
>> available=15942MB].
>>
>> Please change
>>
>> MemoryConfiguration.systemCacheMaxSize
>> and
>>
>> MemoryConfiguration.defaultMemoryPolicySize
>> to decrease
>>
>> memory
>>
>> allocated
>>
>> for each node.
>>
>> Current settings:
>> HeapInit=250MB
>> HeapMax=3543MB
>>
>> DefaultMemoryPolicySize=12753MB
>>
>> SystemCacheInitialSize=40MB
>> SystemCacheMaxSize=100MB
>>
>> Other ignite instanses
>> on the server require:
>> 12853MB
>>
>> I think it's make
>> sense to describe what
>> these numbers are
>>
>> consist
>>
>> of.
>>
>> We simple say which
>> parameters have an
>> impact on how much
>>
>> memory
>>
>> the
>>
>> instance needs and
>> their (parameters)
>> actual values.
>>
>> Also we notice that
>> more than one Ignite
>> instance are ran on
>>
>> the
>>
>> server
>>
>> or
>>
>> workstation and it
>> also consumes memory.
>>
>> 25 авг. 2017 г. 21:30
>> пользователь "Dmitriy
>> Setrakyan" <
>> dsetrakyan@apache.org
>> <mailto:
>> dsetrakyan@apache.org>>
>> написал:
>>
>> Igor, what is this
>> flood of WARN
>> messaging coming
>> after the
>>
>> text?
>>
>> Are
>>
>> we
>>
>> really going to
>> print this whole
>> thing out?
>>
>> On Fri, Aug 25,
>> 2017 at 9:49 AM,
>> Seliverstov Igor <
>>
>> gvvinblade@gmail.com
>> <ma...@gmail.com>
>>
>> wrote:
>>
>> This message
>> appears on
>> topology
>> change in case the
>>
>> available
>>
>> memory
>>
>> is
>>
>> exceeded
>>
>> 2017-08-25
>> 19:47
>> GMT+03:00
>> Seliverstov Igor <
>>
>> gvvinblade@gmail.com
>> <ma...@gmail.com>
>>
>> :
>>
>> An example
>> of current
>> impl:
>>
>>
>> [2017-08-25
>>
>> 19:44:37,740][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> [2017-08-25
>>
>> 19:44:37,740][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> Not enough
>>
>> memory
>>
>> for
>>
>> current
>> process
>>
>> [required=29251MB,
>>
>> available=15942MB].
>> [2017-08-25
>>
>> 19:44:37,740][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> Please
>>
>> change
>>
>>
>> MemoryConfiguration.systemCacheMaxSize
>> and
>>
>> MemoryConfiguration.
>>
>>
>> defaultMemoryPolicySize
>>
>> to
>> decrease
>> memory
>> allocated
>> for each node.
>> [2017-08-25
>>
>> 19:44:37,740][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> [2017-08-25
>>
>> 19:44:37,740][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> Current
>>
>> settings:
>>
>> [2017-08-25
>>
>> 19:44:37,740][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>
>> HeapInit=250MB
>>
>> [2017-08-25
>>
>> 19:44:37,741][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>
>> HeapMax=3543MB
>>
>> [2017-08-25
>>
>> 19:44:37,741][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>
>>
>> DefaultMemoryPolicySize=
>>
>> 12753MB
>> [2017-08-25
>>
>> 19:44:37,741][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>
>> SystemCacheInitialSize=40MB
>> [2017-08-25
>>
>> 19:44:37,741][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>
>>
>> SystemCacheMaxSize=100MB
>>
>> [2017-08-25
>>
>> 19:44:37,741][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> [2017-08-25
>>
>> 19:44:37,741][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> Other
>>
>> ignite
>>
>> instanses
>>
>> on
>>
>> the server
>> require:
>> 12853MB
>> [2017-08-25
>>
>> 19:44:37,741][WARN
>>
>> ][disco-event-worker-#29%
>>
>> internal.
>>
>>
>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>
>>
>> 2017-08-25
>> 17:40
>> GMT+03:00
>> Sergey
>> Kozlov <
>>
>> skozlov@gridgain.com <mailto:skozlov@gridgain.com
>> >
>>
>> :
>>
>> I
>> suppose
>> we
>> should
>> not
>> forget
>> JVM
>> heap
>> size and
>>
>> suggest to
>>
>> reduce
>>
>> both
>>
>> options
>>
>> On
>> Fri,
>> Aug
>> 25,
>> 2017
>> at
>> 5:24
>> PM,
>> Dmitriy
>> Setrakyan
>> <
>>
>>
>> dsetrakyan@apache.org
>> <mailto:
>> dsetrakyan@apache.org>
>>
>> wrote:
>>
>> Igor,
>> I
>> would
>> change
>> the
>>
>> message.
>> How
>> about
>> this:
>>
>>
>> Required
>> RAM
>> size
>> is
>> larger
>> than
>> total
>>
>> physical
>> memory
>>
>> available
>>
>> for
>>
>> OS.
>>
>>
>> Please
>>
>> change
>>
>> MemoryConfiguration.WhichProperty
>>
>> and
>>
>> MemoryPolicyConfiguration.WhichProperty
>> to
>>
>> decrease
>>
>> memory
>>
>> allocated
>>
>> for
>>
>>
>> each
>>
>> node.
>>
>>
>> Also,
>> can
>> we
>>
>> calculate
>> what
>> the
>> memory
>> size
>>
>> allocated
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by 李玉...@163,
18...@163.com.
Currently,the latest version is apache-ignite-fabric-2.1.0-bin.zip
<http://mirrors.hust.edu.cn/apache//ignite/2.1.0/apache-ignite-fabric-2.1.0-bin.zip>.
and in
http://www.gridgainsystems.com/nexus/content/repositories/external/org/apache/ignite/ignite-core/
the latest version is 2.1.4,
As I understand it, add 1 on the version number of 2.1.4, and then
publish it to the ignite main website to ensure that both versions are
consistent,is it ok?
在 2017/9/1 下午6:57, Anton Vinogradov 写道:
> Hi,
>
> Reasonable question, but I propose to keep 2.2 as a release version
> for this urgent release, since we never released X.Y.Z version before.
>
> Let's discuss this at separate thread.
> Could you please start it?
>
>
> On Fri, Sep 1, 2017 at 4:49 AM, 李玉珏@163 <18624049226@163.com
> <ma...@163.com>> wrote:
>
> Since it's an urgent bugfix version, why can't the version number
> be defined as 2.1.1 or 2.1.4?
> After all, functionality has not increased and documents do not
> require big changes.
> If the version number is 2.1 series, there is no need to publish a
> new version of the document.
>
>
>
> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
>
> Igniters,
>
> Seems 2.2 is a urgent bugfix release, so it should be based on
> 2.1,
> In this case all other issues with fixVersion = 2.2 should be
> moved to 2.3.
>
> Currently, I see 835 issues with fixVersion = 2.2
>
> Seems we should have only 4 issues with fixVersion = 2.2:
> - Change default max memory size from 80% to 20% -
> https://issues.apache.org/jira/browse/IGNITE-6182
> <https://issues.apache.org/jira/browse/IGNITE-6182>
> - Detecting low memory on ignite node startup -
> https://issues.apache.org/jira/browse/IGNITE-6003
> <https://issues.apache.org/jira/browse/IGNITE-6003>
> - Backport of improvements to checkpoint algorithm -
> https://issues.apache.org/jira/browse/IGNITE-??
> <https://issues.apache.org/jira/browse/IGNITE-??>??
> - ML profile is missing in 2.1 binary release -
> https://issues.apache.org/jira/browse/IGNITE-6193
> <https://issues.apache.org/jira/browse/IGNITE-6193>
>
> Please correct me in case I've missed something.
>
> Ivan,
>
> Let's include to the release tickets with optimizations of
> checkpointing
>
> algorithm:
>
> https://issues.apache.org/jira/browse/IGNITE-6178
> <https://issues.apache.org/jira/browse/IGNITE-6178>
> https://issues.apache.org/jira/browse/IGNITE-6033
> <https://issues.apache.org/jira/browse/IGNITE-6033>
> https://issues.apache.org/jira/browse/IGNITE-5961
> <https://issues.apache.org/jira/browse/IGNITE-5961>
> This will help users who are experiencing problems with
> slow checkpoints.
> Also, let's include
> https://issues.apache.org/jira/browse/IGNITE-6183
> <https://issues.apache.org/jira/browse/IGNITE-6183> to
>
> soften "Ignite node crashed in the middle of checkpoint"
> message as per
> discussion <
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
> <http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html>
>
> .
>
> All of these issues should have fixVersion 2.3.
> Please create backport issue and link it to all necessary issues.
>
> P.s. I've created branch ignite-2.2, currently it equals to
> ignite-2.1.
>
> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov
> <avinogradov@gridgain.com <ma...@gridgain.com>>
> wrote:
>
> Denis,
>
> BTW, who is considered to be the release manager of
> this release?
>
> I'll do it.
>
> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor
> <gvvinblade@gmail.com <ma...@gmail.com>>
> wrote:
>
> Ok, the check happens at the node start time or on
> NODE_JOIN event
>
> in general it looks like:
>
> 1) calculate expected used memory = heap max + system
> cache max + all
> custom policies max + default policy size and put it
> into a node attribute
>
> 2) get total physycal memory, calculate expected safe
> to be used memory
> amount (leave 4 gb min or 20% of available memory for OS)
>
> 3) if expected used memory + expected used memory of
> other nodes on the
> host > than safe to be used memory amount, start
> calculating suggestions
>
> 4) Each ignite instance needs at least 512mb heap +
> 40mb system cache +
> 100mb default polycy, if available memory is less we
> cannot suggest
> anything reasonable, print warning, stop calculation.
>
> 5) check heap size (shouldn't exceed 30% of available
> memory (total_memory
> - reserved for OS memory) * 30% for all JVMs, if it
> exeedes, suggest just
> calculated value or 512MB minimal)
>
> 6) check if system cache size changed, suggest default
> value if it's so
>
> 7) in case 100 mb * policies count < available memory,
> suggest using
> default policy with max size equals to remaining
> memory (available - heap
> -
> system cache)
>
> 8) calculate new size for each memory policy ( it's
> user defined size *
> (remaining / (all_policies_size * nodes_cnt)); in
> proportion to
> remaining memory, devided by nodes number on the host
> or 100 mb minimal)
>
> 9) print suggestions
>
>
>
> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan
> <dsetrakyan@apache.org <ma...@apache.org>>:
>
> Igor, can you please describe the algorithm with
> all the thresholds?
>
> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor
> <gvvinblade@gmail.com <ma...@gmail.com>
>
> wrote:
>
> The suggestion here is based on initial
> settings, and it's so because
>
> there
>
> is no other nodes on the host in the example.
>
> The algorithm tries to preserve the original
> ratio of memory policies
> keeping numbers reasonable (for example after
> some thresshold it will
> suggest not to use several memory policies if
> there is not enough of
>
> memory
>
> for all of them) and taking into consideration
> nodes count on the
>
> host,
>
> each jvm heap, needed memory for OS, etc
>
> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan
> <dsetrakyan@apache.org
> <ma...@apache.org>>:
>
> Looks good, but why in the example
> provided are we suggesting 8GB? 2
>
> nodes
>
> with 8GB will completely exhaust the
> available memory. I would
>
> suggest
>
> 6
>
> or
>
> 7GB.
>
> Also, why 100MB for default policy.
> Anything under 1GB seems too
>
> small.
>
> Can you please comment?
>
> D.
>
> On Mon, Aug 28, 2017 at 3:31 AM,
> Seliverstov Igor <
>
> gvvinblade@gmail.com <ma...@gmail.com>>
>
> wrote:
>
> One more example of possible warning:
>
> -----------------------------------------------------
> Excessive memory usage by Ignite node
> process (performance may
>
> drop)
>
> [requested=44613MB, available=15942MB].
>
> Please tune the folowing settings as
> suggested:
>
> MemoryPolicyConfiguration.initialSize
> for bigPlc: 8102MB
> MemoryPolicyConfiguration.maxSize
> for bigPlc: 8102MB
>
> MemoryPolicyConfiguration.initialSize
> for dfltPlc: 100MB
> MemoryPolicyConfiguration.maxSize
> for dfltPlc: 100MB
>
> Current settings:
> Java Heap maxSize: 3543MB
> Java Heap initSize: 250MB
>
> MemoryPolicyConfiguration.initialSize
> for bigPlc: 256MB
> MemoryPolicyConfiguration.maxSize
> for bigPlc: 40960MB
>
> MemoryPolicyConfiguration.initialSize
> for dfltPlc: 10MB
> MemoryPolicyConfiguration.maxSize
> for dfltPlc: 10MB
> The overall expected memory usage
> by all Ignite nodes on the
>
> host:
>
> 44613MB
> -----------------------------------------------------
>
> Your thoughts?
>
> 2017-08-28 5:06 GMT+03:00 Denis Magda
> <dmagda@apache.org
> <ma...@apache.org>>:
>
> Guys,
>
> ML lib profile is missing in 2.1
> release! That must be fixed and
>
> rolled
>
> out in this emergency release:
> https://issues.apache.org/jira/browse/IGNITE-6193
> <https://issues.apache.org/jira/browse/IGNITE-6193>
> <
> https://issues.apache.org/jira/browse/IGNITE-6193
> <https://issues.apache.org/jira/browse/IGNITE-6193>>
>
> Oleg, Yuri, please step in and
> handle the issue.
>
> BTW, who is considered to be the
> release manager of this
>
> release?
>
> —
> Denis
>
> On Aug 25, 2017, at 2:29 PM,
> Dmitriy Setrakyan <
>
> dsetrakyan@apache.org
> <ma...@apache.org>>
>
> wrote:
>
> I like the format proposed by
> Denis, very clear.
>
> However, I also do not
> understand why a user should
> change the
>
> size
>
> of
>
> some
>
> system cache. How would a user
> ever know what value to put
>
> there?
>
> This
>
> value should be configured by
> Ignite automatically.
>
> D.
>
> On Fri, Aug 25, 2017 at 2:24
> PM, Denis Magda <
>
> dmagda@apache.org <ma...@apache.org>>
>
> wrote:
>
> Igor,
>
> Let me suggest this format.
>
> ---------------------------------------------
> Excessive memory usage by
> Ignite node process
> (performance
>
> may
>
> drop)
>
> [requested=29251MB,
> available=15942MB]
>
> Please tune the following
> settings:
>
> [MemoryConfiguration.defaultMemoryPolicySize
> = suggested
>
> value]
>
>
> MemoryConfiguration.systemCacheMaxSize
> = suggested value
>
> [MemoryPolicyConfiguration.maxSize
> for {policy_name_1} =
>
> suggested
>
> value]
>
> [MemoryPolicyConfiguration.maxSize
> for {policy_name_2} =
>
> suggested
>
> value]
>
> Current settings:
>
> [DefaultMemoryPolicySize
> = value]
> [{policy_name_1} size =
> value]
> [{policy_name_1} size =
> value]
> SystemCacheInitialSize
> = value
> SystemCacheMaxSize = value
> Java Heap Init Size = value
> Java Heap Max Size = value
>
> The overall memory usage
> by all Ignite nodes on the
> host:
>
> value
>
> -------------------------------------------
>
> Records in […] are
> optional. If custom memory
> policy is not
>
> set
>
> or
>
> the
>
> default memory policy is
> overridden the output will
> miss
>
> some of
>
> the
>
> rows.
>
> As for systemCacheMaxSize,
> it should be show ONLY if the
>
> parameter
>
> was
>
> set
>
> explicitly by user code.
> Otherwise, the platform
> should be
>
> wise
>
> enough
>
> to
>
> instantiate it properly
> depending on the host
> memory usage.
>
> —
> Denis
>
> On Aug 25, 2017, at
> 1:49 PM, Seliverstov
> Igor <
>
> gvvinblade@gmail.com
> <ma...@gmail.com>>
>
> wrote:
>
> The message without
> logging layout:
>
> Not enough memory for
> current process
> [required=29251MB,
>
> available=15942MB].
>
> Please change
> MemoryConfiguration.systemCacheMaxSize
> and
> MemoryConfiguration.defaultMemoryPolicySize
> to decrease
>
> memory
>
> allocated
>
> for each node.
>
> Current settings:
> HeapInit=250MB
> HeapMax=3543MB
>
> DefaultMemoryPolicySize=12753MB
>
> SystemCacheInitialSize=40MB
> SystemCacheMaxSize=100MB
>
> Other ignite instanses
> on the server require:
> 12853MB
>
> I think it's make
> sense to describe what
> these numbers are
>
> consist
>
> of.
>
> We simple say which
> parameters have an
> impact on how much
>
> memory
>
> the
>
> instance needs and
> their (parameters)
> actual values.
>
> Also we notice that
> more than one Ignite
> instance are ran on
>
> the
>
> server
>
> or
>
> workstation and it
> also consumes memory.
>
> 25 авг. 2017 г. 21:30
> пользователь "Dmitriy
> Setrakyan" <
> dsetrakyan@apache.org
> <ma...@apache.org>>
> написал:
>
> Igor, what is this
> flood of WARN
> messaging coming
> after the
>
> text?
>
> Are
>
> we
>
> really going to
> print this whole
> thing out?
>
> On Fri, Aug 25,
> 2017 at 9:49 AM,
> Seliverstov Igor <
>
> gvvinblade@gmail.com
> <ma...@gmail.com>
>
> wrote:
>
> This message
> appears on
> topology
> change in case the
>
> available
>
> memory
>
> is
>
> exceeded
>
> 2017-08-25
> 19:47
> GMT+03:00
> Seliverstov Igor <
>
> gvvinblade@gmail.com
> <ma...@gmail.com>
>
> :
>
> An example
> of current
> impl:
>
>
> [2017-08-25
> 19:44:37,740][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25
> 19:44:37,740][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
> Not enough
>
> memory
>
> for
>
> current
> process
> [required=29251MB,
> available=15942MB].
> [2017-08-25
> 19:44:37,740][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
> Please
>
> change
>
> MemoryConfiguration.systemCacheMaxSize
> and
>
> MemoryConfiguration.
>
> defaultMemoryPolicySize
>
> to
> decrease
> memory
> allocated
> for each node.
> [2017-08-25
> 19:44:37,740][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25
> 19:44:37,740][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
> Current
>
> settings:
>
> [2017-08-25
> 19:44:37,740][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
>
> HeapInit=250MB
>
> [2017-08-25
> 19:44:37,741][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
>
> HeapMax=3543MB
>
> [2017-08-25
> 19:44:37,741][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
>
> DefaultMemoryPolicySize=
>
> 12753MB
> [2017-08-25
> 19:44:37,741][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
> SystemCacheInitialSize=40MB
> [2017-08-25
> 19:44:37,741][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
>
> SystemCacheMaxSize=100MB
>
> [2017-08-25
> 19:44:37,741][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25
> 19:44:37,741][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
> Other
>
> ignite
>
> instanses
>
> on
>
> the server
> require:
> 12853MB
> [2017-08-25
> 19:44:37,741][WARN
> ][disco-event-worker-#29%
>
> internal.
>
> GridHomePathSelfTest0%][GridDiscoveryManager]
>
>
> 2017-08-25
> 17:40
> GMT+03:00
> Sergey
> Kozlov <
>
> skozlov@gridgain.com <ma...@gridgain.com>
>
> :
>
> I
> suppose
> we
> should
> not
> forget
> JVM
> heap
> size and
>
> suggest to
>
> reduce
>
> both
>
> options
>
> On
> Fri,
> Aug
> 25,
> 2017
> at
> 5:24
> PM,
> Dmitriy
> Setrakyan
> <
>
> dsetrakyan@apache.org
> <ma...@apache.org>
>
> wrote:
>
> Igor,
> I
> would
> change
> the
> message.
> How
> about
> this:
>
> Required
> RAM
> size
> is
> larger
> than
> total
> physical
> memory
>
> available
>
> for
>
> OS.
>
> Please
> change
> MemoryConfiguration.WhichProperty
> and
> MemoryPolicyConfiguration.WhichProperty
> to
> decrease
>
> memory
>
> allocated
>
> for
>
> each
> node.
>
>
> Also,
> can
> we
> calculate
> what
> the
> memory
> size
> allocated
>
> for
>
> each
>
> node
>
> should
>
> be?
> In
> that
> case
> we
> should
> suggest
> it.
>
> D.
>
>
> On
> Fri,
> Aug
> 25,
> 2017
> at
> 7:20
> AM,
> Seliverstov
> Igor <
>
> gvvinblade@gmail.com
> <ma...@gmail.com>
>
> wrote:
>
> What
> do
> you,
> guys
> think
> about
> next
> warning?
>
> [2017-08-25
> 17:17:04,718][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][
>
> GridHomePat
>
> hSelfTest0]
>
> System
> cache's
> MemoryPolicy
> size
> is
> configured
> to
> 40
>
> MB.
>
> Use
>
> MemoryConfiguration.systemCacheMemorySize
> property
> to
>
> change
>
> the
>
> setting.
>
> [2017-08-25
> 17:17:04,718][WARN
> ][test-runner-#1%internal.GridHomePathSelfTest%][
>
> GridHomePat
>
> hSelfTest0]
>
> Required
> RAM
> size
> is
> larger
> than
> total
> physical
>
> memory
>
> available
>
> for
>
> OS.
>
> Check
> your
> configuration
> to
> avoid
> swap
> partition
>
> usage.
>
> Use
> MemoryConfiguration
> and
>
> MemoryPolicyConfiguration
>
> to
>
> change
>
> the
>
> settings.
>
> Physical
> memory
> [required=16397MB,
>
> available=15942MB]
>
> [2017-08-25
> 17:17:04,726][WARN
> ][test-runner-#1%internal.GridHomePathSelfTest%][
>
> GridHomePat
>
> hSelfTest0]
>
> Peer
> class
> loading
> is
> enabled
> (disable
> it
> in
>
> production
>
> for
>
> performance
>
> and
>
> deployment
> consistency
> reasons)
> [2017-08-25
> 17:17:04,726][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][
>
> GridHomePat
>
> hSelfTest0]
>
> Configured
> caches
> [in
> 'sysMemPlc'
> memoryPolicy:
>
> ['ignite-sys-cache']]
>
> [2017-08-25
> 17:17:04,731][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][
>
> GridHomePat
>
> hSelfTest0]
>
> 3-rd
> party
> licenses
> can
> be
> found
> at:
> /home/gvvinblade/projects/igni
>
> te/incubator-ignite/libs/
>
> licenses
>
>
> 2017-08-25
> 13:26
> GMT+03:00
> Yakov
> Zhdanov
> <
>
> yzhdanov@apache.org
> <ma...@apache.org>
>
> :
>
> Agree,
> let's
> release
> new
> version
> including
> tickets
>
> mentioned
>
> by
>
> Denis
>
> and
>
> Ivan.
>
> --Yakov
>
>
>
> --
> Sergey
> Kozlov
> GridGain
> Systems
> www.gridgain.com
> <http://www.gridgain.com>
>
>
>
>
>
>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Denis Magda <dm...@apache.org>.
What’s the “new beta release process”? Haven’t seen anything like that on @dev?
—
Denis
> On Sep 6, 2017, at 9:37 AM, Anton Vinogradov <av...@gridgain.com> wrote:
>
> Denis,
>
> As you can remember, Cos told us that the only option to perform release is
> to perform it in Apache way.
> So, we will be able once new release process beta will be ready.
>
> Also, https://issues.apache.org/jira/browse/IGNITE-6193 was reopened and
> fix in progress now.
>
> On Wed, Sep 6, 2017 at 7:26 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Igniters,
>>
>> This is no longer looks like an urgent release. The overall process
>> lingered for around 2 weeks. When do we start the vote?
>>
>> —
>> Denis
>>
>>> On Sep 5, 2017, at 6:22 AM, Igor Sapego <is...@gridgain.com> wrote:
>>>
>>> Cherry-picked from master. Changed "Fix version" in ticket.
>>>
>>> Best Regards,
>>> Igor
>>>
>>> On Tue, Sep 5, 2017 at 4:15 PM, Anton Vinogradov <
>> avinogradov@gridgain.com>
>>> wrote:
>>>
>>>> Igor,
>>>>
>>>> That's an important fix.
>>>> Please merge it to ignite-2.2.
>>>> Don't forget to change fix version.
>>>>
>>>> On Tue, Sep 5, 2017 at 3:42 PM, Igor Sapego <is...@apache.org> wrote:
>>>>
>>>>> Igniters,
>>>>>
>>>>> There is one more fix I would propose to include into 2.2: [1]. It has
>>>>> been fixed a while ago and it is critical for C++ users that would like
>>>>> to make use of persistence store, as currently they need to have
>>>>> separate Java node just to activate cluster.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> [1] - https://issues.apache.org/jira/browse/IGNITE-5771
>>>>>
>>>>> Best Regards,
>>>>> Igor
>>>>>
>>>>> On Mon, Sep 4, 2017 at 6:54 PM, Anton Vinogradov <
>>>> avinogradov@gridgain.com
>>>>>>
>>>>> wrote:
>>>>>
>>>>>> Igniters,
>>>>>>
>>>>>> All fixes related to release 2.2 merged.
>>>>>>
>>>>>> But,
>>>>>> we still waiting for changes related to new release process (
>>>>>> https://issues.apache.org/jira/browse/IGNITE-5249).
>>>>>> Beta version of "new release process" will be used to release 2.2,
>>>> final
>>>>>> changes will be merged to 2.3.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 1, 2017 at 1:58 PM, Dmitry Pavlov <dp...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, at least for this release I suggest to keep versioning as is and
>>>>> use
>>>>>>> 2.2
>>>>>>>
>>>>>>> Without solved issue with http://apache-ignite-
>>>>>>> developers.2346864.n4.nabble.com/usage-analytics-td19504.html Ignite
>>>>>> will
>>>>>>> not remind users about latest available ignite version. Versioning
>>>>>>> structure change will confuse users for now. But keeping 2.2 will
>>>> allow
>>>>>>> Ignite users to take newest version with urgent fixes.
>>>>>>>
>>>>>>> I think is it important to achieve users to update. But without
>>>>>>> notification server is it complex.
>>>>>>>
>>>>>>> Sincerely,
>>>>>>> Dmitriy Pavlov
>>>>>>>
>>>>>>> пт, 1 сент. 2017 г. в 4:50, 李玉珏@163 <18...@163.com>:
>>>>>>>
>>>>>>>> Since it's an urgent bugfix version, why can't the version number be
>>>>>>>> defined as 2.1.1 or 2.1.4?
>>>>>>>> After all, functionality has not increased and documents do not
>>>>> require
>>>>>>>> big changes.
>>>>>>>> If the version number is 2.1 series, there is no need to publish a
>>>> new
>>>>>>>> version of the document.
>>>>>>>>
>>>>>>>>
>>>>>>>> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
>>>>>>>>> Igniters,
>>>>>>>>>
>>>>>>>>> Seems 2.2 is a urgent bugfix release, so it should be based on
>>>> 2.1,
>>>>>>>>> In this case all other issues with fixVersion = 2.2 should be
>>>> moved
>>>>> to
>>>>>>>> 2.3.
>>>>>>>>>
>>>>>>>>> Currently, I see 835 issues with fixVersion = 2.2
>>>>>>>>>
>>>>>>>>> Seems we should have only 4 issues with fixVersion = 2.2:
>>>>>>>>> - Change default max memory size from 80% to 20% -
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6182
>>>>>>>>> - Detecting low memory on ignite node startup -
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6003
>>>>>>>>> - Backport of improvements to checkpoint algorithm -
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-????
>>>>>>>>> - ML profile is missing in 2.1 binary release -
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193
>>>>>>>>>
>>>>>>>>> Please correct me in case I've missed something.
>>>>>>>>>
>>>>>>>>> Ivan,
>>>>>>>>>> Let's include to the release tickets with optimizations of
>>>>>>>> checkpointing
>>>>>>>>> algorithm:
>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6178
>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6033
>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-5961
>>>>>>>>>> This will help users who are experiencing problems with slow
>>>>>>>> checkpoints.
>>>>>>>>>> Also, let's include https://issues.apache.org/
>>>>>> jira/browse/IGNITE-6183
>>>>>>>> to
>>>>>>>>> soften "Ignite node crashed in the middle of checkpoint" message
>>>> as
>>>>>> per
>>>>>>>>> discussion <
>>>>>>>>> http://apache-ignite-developers.2346864.n4.nabble.
>>>>>>>> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
>>>>>>>> false-td20473.html
>>>>>>>>>> .
>>>>>>>>> All of these issues should have fixVersion 2.3.
>>>>>>>>> Please create backport issue and link it to all necessary issues.
>>>>>>>>>
>>>>>>>>> P.s. I've created branch ignite-2.2, currently it equals to
>>>>>> ignite-2.1.
>>>>>>>>>
>>>>>>>>> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
>>>>>>>> avinogradov@gridgain.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Denis,
>>>>>>>>>>
>>>>>>>>>>> BTW, who is considered to be the release manager of this
>>>> release?
>>>>>>>>>> I'll do it.
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
>>>>>>>> gvvinblade@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ok, the check happens at the node start time or on NODE_JOIN
>>>> event
>>>>>>>>>>>
>>>>>>>>>>> in general it looks like:
>>>>>>>>>>>
>>>>>>>>>>> 1) calculate expected used memory = heap max + system cache max
>>>> +
>>>>>> all
>>>>>>>>>>> custom policies max + default policy size and put it into a node
>>>>>>>> attribute
>>>>>>>>>>>
>>>>>>>>>>> 2) get total physycal memory, calculate expected safe to be used
>>>>>>>> memory
>>>>>>>>>>> amount (leave 4 gb min or 20% of available memory for OS)
>>>>>>>>>>>
>>>>>>>>>>> 3) if expected used memory + expected used memory of other nodes
>>>>> on
>>>>>>>> the
>>>>>>>>>>> host > than safe to be used memory amount, start calculating
>>>>>>>> suggestions
>>>>>>>>>>>
>>>>>>>>>>> 4) Each ignite instance needs at least 512mb heap + 40mb system
>>>>>> cache
>>>>>>>> +
>>>>>>>>>>> 100mb default polycy, if available memory is less we cannot
>>>>> suggest
>>>>>>>>>>> anything reasonable, print warning, stop calculation.
>>>>>>>>>>>
>>>>>>>>>>> 5) check heap size (shouldn't exceed 30% of available memory
>>>>>>>> (total_memory
>>>>>>>>>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes,
>>>>> suggest
>>>>>>>> just
>>>>>>>>>>> calculated value or 512MB minimal)
>>>>>>>>>>>
>>>>>>>>>>> 6) check if system cache size changed, suggest default value if
>>>>> it's
>>>>>>>> so
>>>>>>>>>>>
>>>>>>>>>>> 7) in case 100 mb * policies count < available memory, suggest
>>>>> using
>>>>>>>>>>> default policy with max size equals to remaining memory
>>>>> (available -
>>>>>>>> heap
>>>>>>>>>>> -
>>>>>>>>>>> system cache)
>>>>>>>>>>>
>>>>>>>>>>> 8) calculate new size for each memory policy ( it's user defined
>>>>>> size
>>>>>>>> *
>>>>>>>>>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>>>>>>>>>> remaining memory, devided by nodes number on the host or 100 mb
>>>>>>>> minimal)
>>>>>>>>>>>
>>>>>>>>>>> 9) print suggestions
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org
>>>>>>> :
>>>>>>>>>>>
>>>>>>>>>>>> Igor, can you please describe the algorithm with all the
>>>>>> thresholds?
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The suggestion here is based on initial settings, and it's so
>>>>>>>> because
>>>>>>>>>>>> there
>>>>>>>>>>>>> is no other nodes on the host in the example.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The algorithm tries to preserve the original ratio of memory
>>>>>>>> policies
>>>>>>>>>>>>> keeping numbers reasonable (for example after some thresshold
>>>> it
>>>>>>>> will
>>>>>>>>>>>>> suggest not to use several memory policies if there is not
>>>>> enough
>>>>>> of
>>>>>>>>>>>> memory
>>>>>>>>>>>>> for all of them) and taking into consideration nodes count on
>>>>> the
>>>>>>>>>>> host,
>>>>>>>>>>>>> each jvm heap, needed memory for OS, etc
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org
>>>>>>>>> :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looks good, but why in the example provided are we suggesting
>>>>>> 8GB?
>>>>>>>> 2
>>>>>>>>>>>>> nodes
>>>>>>>>>>>>>> with 8GB will completely exhaust the available memory. I
>>>> would
>>>>>>>>>>> suggest
>>>>>>>>>>>> 6
>>>>>>>>>>>>> or
>>>>>>>>>>>>>> 7GB.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, why 100MB for default policy. Anything under 1GB seems
>>>>> too
>>>>>>>>>>> small.
>>>>>>>>>>>>>> Can you please comment?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>>>>>>>>>>>> gvvinblade@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One more example of possible warning:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>>>> Excessive memory usage by Ignite node process (performance
>>>> may
>>>>>>>>>>> drop)
>>>>>>>>>>>>>>> [requested=44613MB, available=15942MB].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please tune the folowing settings as suggested:
>>>>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>>>>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>>>>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>>>>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>>>> Java Heap maxSize: 3543MB
>>>>>>>>>>>>>>> Java Heap initSize: 250MB
>>>>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>>>>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc:
>>>> 40960MB
>>>>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>>>>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>>>>>>>>>>>>>>> The overall expected memory usage by all Ignite nodes on
>>>>> the
>>>>>>>>>>> host:
>>>>>>>>>>>>>>> 44613MB
>>>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your thoughts?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ML lib profile is missing in 2.1 release! That must be
>>>> fixed
>>>>>> and
>>>>>>>>>>>>> rolled
>>>>>>>>>>>>>>>> out in this emergency release:
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Oleg, Yuri, please step in and handle the issue.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> BTW, who is considered to be the release manager of this
>>>>>>>>>>> release?
>>>>>>>>>>>>>>>> —
>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>> dsetrakyan@apache.org>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> I like the format proposed by Denis, very clear.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, I also do not understand why a user should change
>>>>> the
>>>>>>>>>>>> size
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>> system cache. How would a user ever know what value to put
>>>>>>>>>>> there?
>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>> value should be configured by Ignite automatically.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>>>>>>>>>>> dmagda@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Igor,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Let me suggest this format.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>>>>>>> Excessive memory usage by Ignite node process
>>>> (performance
>>>>>>>>>>> may
>>>>>>>>>>>>> drop)
>>>>>>>>>>>>>>>>>> [requested=29251MB, available=15942MB]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please tune the following settings:
>>>>>>>>>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize =
>>>> suggested
>>>>>>>>>>>> value]
>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested
>>>> value
>>>>>>>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for
>>>> {policy_name_1} =
>>>>>>>>>>>>> suggested
>>>>>>>>>>>>>>>>>> value]
>>>>>>>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for
>>>> {policy_name_2} =
>>>>>>>>>>>>> suggested
>>>>>>>>>>>>>>>>>> value]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>>>>>>> [DefaultMemoryPolicySize = value]
>>>>>>>>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>>>>>>>>> SystemCacheInitialSize = value
>>>>>>>>>>>>>>>>>> SystemCacheMaxSize = value
>>>>>>>>>>>>>>>>>> Java Heap Init Size = value
>>>>>>>>>>>>>>>>>> Java Heap Max Size = value
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Records in […] are optional. If custom memory policy is
>>>> not
>>>>>>>>>>> set
>>>>>>>>>>>> or
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> default memory policy is overridden the output will miss
>>>>>>>>>>> some of
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> rows.
>>>>>>>>>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>>>>>>>>>>>> parameter
>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>> explicitly by user code. Otherwise, the platform should
>>>> be
>>>>>>>>>>> wise
>>>>>>>>>>>>>> enough
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> instantiate it properly depending on the host memory
>>>> usage.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> —
>>>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>>>>>>>>>>>>>> gvvinblade@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> The message without logging layout:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Not enough memory for current process [required=29251MB,
>>>>>>>>>>>>>>>>>> available=15942MB].
>>>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize
>>>> and
>>>>>>>>>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>>>>>>>>>>> memory
>>>>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>>>>>> for each node.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>>>>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think it's make sense to describe what these numbers
>>>> are
>>>>>>>>>>>>> consist
>>>>>>>>>>>>>>> of.
>>>>>>>>>>>>>>>>>>> We simple say which parameters have an impact on how
>>>> much
>>>>>>>>>>>> memory
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> instance needs and their (parameters) actual values.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Also we notice that more than one Ignite instance are
>>>> ran
>>>>> on
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> workstation and it also consumes memory.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>>>>>>>>>>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after
>>>>> the
>>>>>>>>>>>>> text?
>>>>>>>>>>>>>>> Are
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> really going to print this whole thing out?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>>>>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This message appears on topology change in case the
>>>>>>>>>>> available
>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>> An example of current impl:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not
>>>>> enough
>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> current process [required=29251MB,
>>>> available=15942MB].
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>>>> MemoryConfiguration.
>>>>>>>>>>>>>>>>>>>>> defaultMemoryPolicySize
>>>>>>>>>>>>>>>>>>>>>> to decrease memory allocated for each node.
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>> Current
>>>>>>>>>>>>> settings:
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>> DefaultMemoryPolicySize=
>>>>>>>>>>>>>>>>>>>>>> 12753MB
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>>>>>>>>>>> ignite
>>>>>>>>>>>>>>>> instanses
>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>> the server require: 12853MB
>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>>>> ][disco-event-worker-#29%
>>>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>>>>>>>>>>>>> skozlov@gridgain.com
>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
>>>>>>>>>>> suggest to
>>>>>>>>>>>>>>> reduce
>>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>>>> memory
>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty
>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to
>>>> decrease
>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> each node.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Also, can we calculate what the memory size
>>>> allocated
>>>>>>>>>>> for
>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>> be? In that case we should suggest it.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>>>>>>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to
>>>> 40
>>>>>>>>>>> MB.
>>>>>>>>>>>>> Use
>>>>>>>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize
>>>> property
>>>>> to
>>>>>>>>>>>>> change
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> setting.
>>>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap
>>>> partition
>>>>>>>>>>>> usage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
>>>>>>>>>>> MemoryPolicyConfiguration
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> settings.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>>>>>>>>>>> available=15942MB]
>>>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>>>>>>>>>>> production
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> deployment consistency reasons)
>>>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>>>>>>>>>>>>>>>>>> ['ignite-sys-cache']]
>>>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>>>>>>>>>>> te/incubator-ignite/libs/
>>>>>>>>>>>>>>> licenses
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>>>>>>>>>>>>>> yzhdanov@apache.org
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>>>> Agree, let's release new version including
>>>> tickets
>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> Ivan.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --Yakov
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Sergey Kozlov
>>>>>>>>>>>>>>>>>>>>>>> GridGain Systems
>>>>>>>>>>>>>>>>>>>>>>> www.gridgain.com
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Anton Vinogradov <av...@gridgain.com>.
Denis,
As you can remember, Cos told us that the only option to perform release is
to perform it in Apache way.
So, we will be able once new release process beta will be ready.
Also, https://issues.apache.org/jira/browse/IGNITE-6193 was reopened and
fix in progress now.
On Wed, Sep 6, 2017 at 7:26 PM, Denis Magda <dm...@apache.org> wrote:
> Igniters,
>
> This is no longer looks like an urgent release. The overall process
> lingered for around 2 weeks. When do we start the vote?
>
> —
> Denis
>
> > On Sep 5, 2017, at 6:22 AM, Igor Sapego <is...@gridgain.com> wrote:
> >
> > Cherry-picked from master. Changed "Fix version" in ticket.
> >
> > Best Regards,
> > Igor
> >
> > On Tue, Sep 5, 2017 at 4:15 PM, Anton Vinogradov <
> avinogradov@gridgain.com>
> > wrote:
> >
> >> Igor,
> >>
> >> That's an important fix.
> >> Please merge it to ignite-2.2.
> >> Don't forget to change fix version.
> >>
> >> On Tue, Sep 5, 2017 at 3:42 PM, Igor Sapego <is...@apache.org> wrote:
> >>
> >>> Igniters,
> >>>
> >>> There is one more fix I would propose to include into 2.2: [1]. It has
> >>> been fixed a while ago and it is critical for C++ users that would like
> >>> to make use of persistence store, as currently they need to have
> >>> separate Java node just to activate cluster.
> >>>
> >>> What do you think?
> >>>
> >>> [1] - https://issues.apache.org/jira/browse/IGNITE-5771
> >>>
> >>> Best Regards,
> >>> Igor
> >>>
> >>> On Mon, Sep 4, 2017 at 6:54 PM, Anton Vinogradov <
> >> avinogradov@gridgain.com
> >>>>
> >>> wrote:
> >>>
> >>>> Igniters,
> >>>>
> >>>> All fixes related to release 2.2 merged.
> >>>>
> >>>> But,
> >>>> we still waiting for changes related to new release process (
> >>>> https://issues.apache.org/jira/browse/IGNITE-5249).
> >>>> Beta version of "new release process" will be used to release 2.2,
> >> final
> >>>> changes will be merged to 2.3.
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Sep 1, 2017 at 1:58 PM, Dmitry Pavlov <dp...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi, at least for this release I suggest to keep versioning as is and
> >>> use
> >>>>> 2.2
> >>>>>
> >>>>> Without solved issue with http://apache-ignite-
> >>>>> developers.2346864.n4.nabble.com/usage-analytics-td19504.html Ignite
> >>>> will
> >>>>> not remind users about latest available ignite version. Versioning
> >>>>> structure change will confuse users for now. But keeping 2.2 will
> >> allow
> >>>>> Ignite users to take newest version with urgent fixes.
> >>>>>
> >>>>> I think is it important to achieve users to update. But without
> >>>>> notification server is it complex.
> >>>>>
> >>>>> Sincerely,
> >>>>> Dmitriy Pavlov
> >>>>>
> >>>>> пт, 1 сент. 2017 г. в 4:50, 李玉珏@163 <18...@163.com>:
> >>>>>
> >>>>>> Since it's an urgent bugfix version, why can't the version number be
> >>>>>> defined as 2.1.1 or 2.1.4?
> >>>>>> After all, functionality has not increased and documents do not
> >>> require
> >>>>>> big changes.
> >>>>>> If the version number is 2.1 series, there is no need to publish a
> >> new
> >>>>>> version of the document.
> >>>>>>
> >>>>>>
> >>>>>> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
> >>>>>>> Igniters,
> >>>>>>>
> >>>>>>> Seems 2.2 is a urgent bugfix release, so it should be based on
> >> 2.1,
> >>>>>>> In this case all other issues with fixVersion = 2.2 should be
> >> moved
> >>> to
> >>>>>> 2.3.
> >>>>>>>
> >>>>>>> Currently, I see 835 issues with fixVersion = 2.2
> >>>>>>>
> >>>>>>> Seems we should have only 4 issues with fixVersion = 2.2:
> >>>>>>> - Change default max memory size from 80% to 20% -
> >>>>>>> https://issues.apache.org/jira/browse/IGNITE-6182
> >>>>>>> - Detecting low memory on ignite node startup -
> >>>>>>> https://issues.apache.org/jira/browse/IGNITE-6003
> >>>>>>> - Backport of improvements to checkpoint algorithm -
> >>>>>>> https://issues.apache.org/jira/browse/IGNITE-????
> >>>>>>> - ML profile is missing in 2.1 binary release -
> >>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193
> >>>>>>>
> >>>>>>> Please correct me in case I've missed something.
> >>>>>>>
> >>>>>>> Ivan,
> >>>>>>>> Let's include to the release tickets with optimizations of
> >>>>>> checkpointing
> >>>>>>> algorithm:
> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6178
> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6033
> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-5961
> >>>>>>>> This will help users who are experiencing problems with slow
> >>>>>> checkpoints.
> >>>>>>>> Also, let's include https://issues.apache.org/
> >>>> jira/browse/IGNITE-6183
> >>>>>> to
> >>>>>>> soften "Ignite node crashed in the middle of checkpoint" message
> >> as
> >>>> per
> >>>>>>> discussion <
> >>>>>>> http://apache-ignite-developers.2346864.n4.nabble.
> >>>>>> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
> >>>>>> false-td20473.html
> >>>>>>>> .
> >>>>>>> All of these issues should have fixVersion 2.3.
> >>>>>>> Please create backport issue and link it to all necessary issues.
> >>>>>>>
> >>>>>>> P.s. I've created branch ignite-2.2, currently it equals to
> >>>> ignite-2.1.
> >>>>>>>
> >>>>>>> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
> >>>>>> avinogradov@gridgain.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Denis,
> >>>>>>>>
> >>>>>>>>> BTW, who is considered to be the release manager of this
> >> release?
> >>>>>>>> I'll do it.
> >>>>>>>>
> >>>>>>>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
> >>>>>> gvvinblade@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Ok, the check happens at the node start time or on NODE_JOIN
> >> event
> >>>>>>>>>
> >>>>>>>>> in general it looks like:
> >>>>>>>>>
> >>>>>>>>> 1) calculate expected used memory = heap max + system cache max
> >> +
> >>>> all
> >>>>>>>>> custom policies max + default policy size and put it into a node
> >>>>>> attribute
> >>>>>>>>>
> >>>>>>>>> 2) get total physycal memory, calculate expected safe to be used
> >>>>>> memory
> >>>>>>>>> amount (leave 4 gb min or 20% of available memory for OS)
> >>>>>>>>>
> >>>>>>>>> 3) if expected used memory + expected used memory of other nodes
> >>> on
> >>>>>> the
> >>>>>>>>> host > than safe to be used memory amount, start calculating
> >>>>>> suggestions
> >>>>>>>>>
> >>>>>>>>> 4) Each ignite instance needs at least 512mb heap + 40mb system
> >>>> cache
> >>>>>> +
> >>>>>>>>> 100mb default polycy, if available memory is less we cannot
> >>> suggest
> >>>>>>>>> anything reasonable, print warning, stop calculation.
> >>>>>>>>>
> >>>>>>>>> 5) check heap size (shouldn't exceed 30% of available memory
> >>>>>> (total_memory
> >>>>>>>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes,
> >>> suggest
> >>>>>> just
> >>>>>>>>> calculated value or 512MB minimal)
> >>>>>>>>>
> >>>>>>>>> 6) check if system cache size changed, suggest default value if
> >>> it's
> >>>>>> so
> >>>>>>>>>
> >>>>>>>>> 7) in case 100 mb * policies count < available memory, suggest
> >>> using
> >>>>>>>>> default policy with max size equals to remaining memory
> >>> (available -
> >>>>>> heap
> >>>>>>>>> -
> >>>>>>>>> system cache)
> >>>>>>>>>
> >>>>>>>>> 8) calculate new size for each memory policy ( it's user defined
> >>>> size
> >>>>>> *
> >>>>>>>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> >>>>>>>>> remaining memory, devided by nodes number on the host or 100 mb
> >>>>>> minimal)
> >>>>>>>>>
> >>>>>>>>> 9) print suggestions
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <
> >>> dsetrakyan@apache.org
> >>>>> :
> >>>>>>>>>
> >>>>>>>>>> Igor, can you please describe the algorithm with all the
> >>>> thresholds?
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
> >>>>>> gvvinblade@gmail.com
> >>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> The suggestion here is based on initial settings, and it's so
> >>>>>> because
> >>>>>>>>>> there
> >>>>>>>>>>> is no other nodes on the host in the example.
> >>>>>>>>>>>
> >>>>>>>>>>> The algorithm tries to preserve the original ratio of memory
> >>>>>> policies
> >>>>>>>>>>> keeping numbers reasonable (for example after some thresshold
> >> it
> >>>>>> will
> >>>>>>>>>>> suggest not to use several memory policies if there is not
> >>> enough
> >>>> of
> >>>>>>>>>> memory
> >>>>>>>>>>> for all of them) and taking into consideration nodes count on
> >>> the
> >>>>>>>>> host,
> >>>>>>>>>>> each jvm heap, needed memory for OS, etc
> >>>>>>>>>>>
> >>>>>>>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
> >>>> dsetrakyan@apache.org
> >>>>>>> :
> >>>>>>>>>>>
> >>>>>>>>>>>> Looks good, but why in the example provided are we suggesting
> >>>> 8GB?
> >>>>>> 2
> >>>>>>>>>>> nodes
> >>>>>>>>>>>> with 8GB will completely exhaust the available memory. I
> >> would
> >>>>>>>>> suggest
> >>>>>>>>>> 6
> >>>>>>>>>>> or
> >>>>>>>>>>>> 7GB.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Also, why 100MB for default policy. Anything under 1GB seems
> >>> too
> >>>>>>>>> small.
> >>>>>>>>>>>> Can you please comment?
> >>>>>>>>>>>>
> >>>>>>>>>>>> D.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> >>>>>>>>>> gvvinblade@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> One more example of possible warning:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -----------------------------------------------------
> >>>>>>>>>>>>> Excessive memory usage by Ignite node process (performance
> >> may
> >>>>>>>>> drop)
> >>>>>>>>>>>>> [requested=44613MB, available=15942MB].
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Please tune the folowing settings as suggested:
> >>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> >>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> >>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> >>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Current settings:
> >>>>>>>>>>>>> Java Heap maxSize: 3543MB
> >>>>>>>>>>>>> Java Heap initSize: 250MB
> >>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> >>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc:
> >> 40960MB
> >>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> >>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> >>>>>>>>>>>>> The overall expected memory usage by all Ignite nodes on
> >>> the
> >>>>>>>>> host:
> >>>>>>>>>>>>> 44613MB
> >>>>>>>>>>>>> -----------------------------------------------------
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Your thoughts?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Guys,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ML lib profile is missing in 2.1 release! That must be
> >> fixed
> >>>> and
> >>>>>>>>>>> rolled
> >>>>>>>>>>>>>> out in this emergency release:
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Oleg, Yuri, please step in and handle the issue.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> BTW, who is considered to be the release manager of this
> >>>>>>>>> release?
> >>>>>>>>>>>>>> —
> >>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>> dsetrakyan@apache.org>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> I like the format proposed by Denis, very clear.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> However, I also do not understand why a user should change
> >>> the
> >>>>>>>>>> size
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>> system cache. How would a user ever know what value to put
> >>>>>>>>> there?
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>>> value should be configured by Ignite automatically.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> >>>>>>>>> dmagda@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>> Igor,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Let me suggest this format.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ---------------------------------------------
> >>>>>>>>>>>>>>>> Excessive memory usage by Ignite node process
> >> (performance
> >>>>>>>>> may
> >>>>>>>>>>> drop)
> >>>>>>>>>>>>>>>> [requested=29251MB, available=15942MB]
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Please tune the following settings:
> >>>>>>>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize =
> >> suggested
> >>>>>>>>>> value]
> >>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested
> >> value
> >>>>>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for
> >> {policy_name_1} =
> >>>>>>>>>>> suggested
> >>>>>>>>>>>>>>>> value]
> >>>>>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for
> >> {policy_name_2} =
> >>>>>>>>>>> suggested
> >>>>>>>>>>>>>>>> value]
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Current settings:
> >>>>>>>>>>>>>>>> [DefaultMemoryPolicySize = value]
> >>>>>>>>>>>>>>>> [{policy_name_1} size = value]
> >>>>>>>>>>>>>>>> [{policy_name_1} size = value]
> >>>>>>>>>>>>>>>> SystemCacheInitialSize = value
> >>>>>>>>>>>>>>>> SystemCacheMaxSize = value
> >>>>>>>>>>>>>>>> Java Heap Init Size = value
> >>>>>>>>>>>>>>>> Java Heap Max Size = value
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
> >>>>>>>>> value
> >>>>>>>>>>>>>>>> -------------------------------------------
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Records in […] are optional. If custom memory policy is
> >> not
> >>>>>>>>> set
> >>>>>>>>>> or
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> default memory policy is overridden the output will miss
> >>>>>>>>> some of
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> rows.
> >>>>>>>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
> >>>>>>>>>> parameter
> >>>>>>>>>>>> was
> >>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>> explicitly by user code. Otherwise, the platform should
> >> be
> >>>>>>>>> wise
> >>>>>>>>>>>> enough
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> instantiate it properly depending on the host memory
> >> usage.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> —
> >>>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> >>>>>>>>>>>> gvvinblade@gmail.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>> The message without logging layout:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Not enough memory for current process [required=29251MB,
> >>>>>>>>>>>>>>>> available=15942MB].
> >>>>>>>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize
> >> and
> >>>>>>>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> >>>>>>>>> memory
> >>>>>>>>>>>>>> allocated
> >>>>>>>>>>>>>>>>> for each node.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Current settings:
> >>>>>>>>>>>>>>>>> HeapInit=250MB
> >>>>>>>>>>>>>>>>> HeapMax=3543MB
> >>>>>>>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
> >>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> >>>>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think it's make sense to describe what these numbers
> >> are
> >>>>>>>>>>> consist
> >>>>>>>>>>>>> of.
> >>>>>>>>>>>>>>>>> We simple say which parameters have an impact on how
> >> much
> >>>>>>>>>> memory
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> instance needs and their (parameters) actual values.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Also we notice that more than one Ignite instance are
> >> ran
> >>> on
> >>>>>>>>>> the
> >>>>>>>>>>>>> server
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> workstation and it also consumes memory.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> >>>>>>>>>>>>>>>>> dsetrakyan@apache.org> написал:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after
> >>> the
> >>>>>>>>>>> text?
> >>>>>>>>>>>>> Are
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> really going to print this whole thing out?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> >>>>>>>>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> This message appears on topology change in case the
> >>>>>>>>> available
> >>>>>>>>>>>>> memory
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> exceeded
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> >>>>>>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>> An example of current impl:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not
> >>> enough
> >>>>>>>>>>> memory
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> current process [required=29251MB,
> >> available=15942MB].
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> >>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
> >>>>>>>>>>> MemoryConfiguration.
> >>>>>>>>>>>>>>>>>>> defaultMemoryPolicySize
> >>>>>>>>>>>>>>>>>>>> to decrease memory allocated for each node.
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> Current
> >>>>>>>>>>> settings:
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>> HeapInit=250MB
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>> HeapMax=3543MB
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>>> DefaultMemoryPolicySize=
> >>>>>>>>>>>>>>>>>>>> 12753MB
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> >>>>>>>>> ignite
> >>>>>>>>>>>>>> instanses
> >>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>> the server require: 12853MB
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> >>> ][disco-event-worker-#29%
> >>>>>>>>>>>> internal.
> >>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> >>>>>>>>>>> skozlov@gridgain.com
> >>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
> >>>>>>>>> suggest to
> >>>>>>>>>>>>> reduce
> >>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>> options
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>>>>>>>>> dsetrakyan@apache.org
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> >>> memory
> >>>>>>>>>>>> available
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> OS.
> >>>>>>>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty
> >> and
> >>>>>>>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to
> >> decrease
> >>>>>>>>>> memory
> >>>>>>>>>>>>>>>>>>> allocated
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> each node.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Also, can we calculate what the memory size
> >> allocated
> >>>>>>>>> for
> >>>>>>>>>>> each
> >>>>>>>>>>>>>> node
> >>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>> be? In that case we should suggest it.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> >>>>>>>>>>>>>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
> >>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to
> >> 40
> >>>>>>>>> MB.
> >>>>>>>>>>> Use
> >>>>>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize
> >> property
> >>> to
> >>>>>>>>>>> change
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> setting.
> >>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
> >>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> >>>>>>>>> memory
> >>>>>>>>>>>>>>>>>> available
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> OS.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap
> >> partition
> >>>>>>>>>> usage.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
> >>>>>>>>> MemoryPolicyConfiguration
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> settings.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
> >>>>>>>>> available=15942MB]
> >>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
> >>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
> >>>>>>>>> production
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> performance
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>> deployment consistency reasons)
> >>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
> >>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> >>>>>>>>>>>>>>>>>>> ['ignite-sys-cache']]
> >>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
> >>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
> >>>>>>>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
> >>>>>>>>> te/incubator-ignite/libs/
> >>>>>>>>>>>>> licenses
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> >>>>>>>>>>>> yzhdanov@apache.org
> >>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>> Agree, let's release new version including
> >> tickets
> >>>>>>>>>>> mentioned
> >>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> Ivan.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> --Yakov
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>> Sergey Kozlov
> >>>>>>>>>>>>>>>>>>>>> GridGain Systems
> >>>>>>>>>>>>>>>>>>>>> www.gridgain.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Denis Magda <dm...@apache.org>.
Igniters,
This is no longer looks like an urgent release. The overall process lingered for around 2 weeks. When do we start the vote?
—
Denis
> On Sep 5, 2017, at 6:22 AM, Igor Sapego <is...@gridgain.com> wrote:
>
> Cherry-picked from master. Changed "Fix version" in ticket.
>
> Best Regards,
> Igor
>
> On Tue, Sep 5, 2017 at 4:15 PM, Anton Vinogradov <av...@gridgain.com>
> wrote:
>
>> Igor,
>>
>> That's an important fix.
>> Please merge it to ignite-2.2.
>> Don't forget to change fix version.
>>
>> On Tue, Sep 5, 2017 at 3:42 PM, Igor Sapego <is...@apache.org> wrote:
>>
>>> Igniters,
>>>
>>> There is one more fix I would propose to include into 2.2: [1]. It has
>>> been fixed a while ago and it is critical for C++ users that would like
>>> to make use of persistence store, as currently they need to have
>>> separate Java node just to activate cluster.
>>>
>>> What do you think?
>>>
>>> [1] - https://issues.apache.org/jira/browse/IGNITE-5771
>>>
>>> Best Regards,
>>> Igor
>>>
>>> On Mon, Sep 4, 2017 at 6:54 PM, Anton Vinogradov <
>> avinogradov@gridgain.com
>>>>
>>> wrote:
>>>
>>>> Igniters,
>>>>
>>>> All fixes related to release 2.2 merged.
>>>>
>>>> But,
>>>> we still waiting for changes related to new release process (
>>>> https://issues.apache.org/jira/browse/IGNITE-5249).
>>>> Beta version of "new release process" will be used to release 2.2,
>> final
>>>> changes will be merged to 2.3.
>>>>
>>>>
>>>>
>>>> On Fri, Sep 1, 2017 at 1:58 PM, Dmitry Pavlov <dp...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, at least for this release I suggest to keep versioning as is and
>>> use
>>>>> 2.2
>>>>>
>>>>> Without solved issue with http://apache-ignite-
>>>>> developers.2346864.n4.nabble.com/usage-analytics-td19504.html Ignite
>>>> will
>>>>> not remind users about latest available ignite version. Versioning
>>>>> structure change will confuse users for now. But keeping 2.2 will
>> allow
>>>>> Ignite users to take newest version with urgent fixes.
>>>>>
>>>>> I think is it important to achieve users to update. But without
>>>>> notification server is it complex.
>>>>>
>>>>> Sincerely,
>>>>> Dmitriy Pavlov
>>>>>
>>>>> пт, 1 сент. 2017 г. в 4:50, 李玉珏@163 <18...@163.com>:
>>>>>
>>>>>> Since it's an urgent bugfix version, why can't the version number be
>>>>>> defined as 2.1.1 or 2.1.4?
>>>>>> After all, functionality has not increased and documents do not
>>> require
>>>>>> big changes.
>>>>>> If the version number is 2.1 series, there is no need to publish a
>> new
>>>>>> version of the document.
>>>>>>
>>>>>>
>>>>>> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
>>>>>>> Igniters,
>>>>>>>
>>>>>>> Seems 2.2 is a urgent bugfix release, so it should be based on
>> 2.1,
>>>>>>> In this case all other issues with fixVersion = 2.2 should be
>> moved
>>> to
>>>>>> 2.3.
>>>>>>>
>>>>>>> Currently, I see 835 issues with fixVersion = 2.2
>>>>>>>
>>>>>>> Seems we should have only 4 issues with fixVersion = 2.2:
>>>>>>> - Change default max memory size from 80% to 20% -
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6182
>>>>>>> - Detecting low memory on ignite node startup -
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6003
>>>>>>> - Backport of improvements to checkpoint algorithm -
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-????
>>>>>>> - ML profile is missing in 2.1 binary release -
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193
>>>>>>>
>>>>>>> Please correct me in case I've missed something.
>>>>>>>
>>>>>>> Ivan,
>>>>>>>> Let's include to the release tickets with optimizations of
>>>>>> checkpointing
>>>>>>> algorithm:
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6178
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6033
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-5961
>>>>>>>> This will help users who are experiencing problems with slow
>>>>>> checkpoints.
>>>>>>>> Also, let's include https://issues.apache.org/
>>>> jira/browse/IGNITE-6183
>>>>>> to
>>>>>>> soften "Ignite node crashed in the middle of checkpoint" message
>> as
>>>> per
>>>>>>> discussion <
>>>>>>> http://apache-ignite-developers.2346864.n4.nabble.
>>>>>> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
>>>>>> false-td20473.html
>>>>>>>> .
>>>>>>> All of these issues should have fixVersion 2.3.
>>>>>>> Please create backport issue and link it to all necessary issues.
>>>>>>>
>>>>>>> P.s. I've created branch ignite-2.2, currently it equals to
>>>> ignite-2.1.
>>>>>>>
>>>>>>> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
>>>>>> avinogradov@gridgain.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Denis,
>>>>>>>>
>>>>>>>>> BTW, who is considered to be the release manager of this
>> release?
>>>>>>>> I'll do it.
>>>>>>>>
>>>>>>>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
>>>>>> gvvinblade@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ok, the check happens at the node start time or on NODE_JOIN
>> event
>>>>>>>>>
>>>>>>>>> in general it looks like:
>>>>>>>>>
>>>>>>>>> 1) calculate expected used memory = heap max + system cache max
>> +
>>>> all
>>>>>>>>> custom policies max + default policy size and put it into a node
>>>>>> attribute
>>>>>>>>>
>>>>>>>>> 2) get total physycal memory, calculate expected safe to be used
>>>>>> memory
>>>>>>>>> amount (leave 4 gb min or 20% of available memory for OS)
>>>>>>>>>
>>>>>>>>> 3) if expected used memory + expected used memory of other nodes
>>> on
>>>>>> the
>>>>>>>>> host > than safe to be used memory amount, start calculating
>>>>>> suggestions
>>>>>>>>>
>>>>>>>>> 4) Each ignite instance needs at least 512mb heap + 40mb system
>>>> cache
>>>>>> +
>>>>>>>>> 100mb default polycy, if available memory is less we cannot
>>> suggest
>>>>>>>>> anything reasonable, print warning, stop calculation.
>>>>>>>>>
>>>>>>>>> 5) check heap size (shouldn't exceed 30% of available memory
>>>>>> (total_memory
>>>>>>>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes,
>>> suggest
>>>>>> just
>>>>>>>>> calculated value or 512MB minimal)
>>>>>>>>>
>>>>>>>>> 6) check if system cache size changed, suggest default value if
>>> it's
>>>>>> so
>>>>>>>>>
>>>>>>>>> 7) in case 100 mb * policies count < available memory, suggest
>>> using
>>>>>>>>> default policy with max size equals to remaining memory
>>> (available -
>>>>>> heap
>>>>>>>>> -
>>>>>>>>> system cache)
>>>>>>>>>
>>>>>>>>> 8) calculate new size for each memory policy ( it's user defined
>>>> size
>>>>>> *
>>>>>>>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>>>>>>>> remaining memory, devided by nodes number on the host or 100 mb
>>>>>> minimal)
>>>>>>>>>
>>>>>>>>> 9) print suggestions
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <
>>> dsetrakyan@apache.org
>>>>> :
>>>>>>>>>
>>>>>>>>>> Igor, can you please describe the algorithm with all the
>>>> thresholds?
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
>>>>>> gvvinblade@gmail.com
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> The suggestion here is based on initial settings, and it's so
>>>>>> because
>>>>>>>>>> there
>>>>>>>>>>> is no other nodes on the host in the example.
>>>>>>>>>>>
>>>>>>>>>>> The algorithm tries to preserve the original ratio of memory
>>>>>> policies
>>>>>>>>>>> keeping numbers reasonable (for example after some thresshold
>> it
>>>>>> will
>>>>>>>>>>> suggest not to use several memory policies if there is not
>>> enough
>>>> of
>>>>>>>>>> memory
>>>>>>>>>>> for all of them) and taking into consideration nodes count on
>>> the
>>>>>>>>> host,
>>>>>>>>>>> each jvm heap, needed memory for OS, etc
>>>>>>>>>>>
>>>>>>>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
>>>> dsetrakyan@apache.org
>>>>>>> :
>>>>>>>>>>>
>>>>>>>>>>>> Looks good, but why in the example provided are we suggesting
>>>> 8GB?
>>>>>> 2
>>>>>>>>>>> nodes
>>>>>>>>>>>> with 8GB will completely exhaust the available memory. I
>> would
>>>>>>>>> suggest
>>>>>>>>>> 6
>>>>>>>>>>> or
>>>>>>>>>>>> 7GB.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, why 100MB for default policy. Anything under 1GB seems
>>> too
>>>>>>>>> small.
>>>>>>>>>>>> Can you please comment?
>>>>>>>>>>>>
>>>>>>>>>>>> D.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>>>>>>>>>> gvvinblade@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> One more example of possible warning:
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>> Excessive memory usage by Ignite node process (performance
>> may
>>>>>>>>> drop)
>>>>>>>>>>>>> [requested=44613MB, available=15942MB].
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please tune the folowing settings as suggested:
>>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>>>>>>>>>>>>>
>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>> Java Heap maxSize: 3543MB
>>>>>>>>>>>>> Java Heap initSize: 250MB
>>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc:
>> 40960MB
>>>>>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>>>>>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>>>>>>>>>>>>> The overall expected memory usage by all Ignite nodes on
>>> the
>>>>>>>>> host:
>>>>>>>>>>>>> 44613MB
>>>>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your thoughts?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ML lib profile is missing in 2.1 release! That must be
>> fixed
>>>> and
>>>>>>>>>>> rolled
>>>>>>>>>>>>>> out in this emergency release:
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Oleg, Yuri, please step in and handle the issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BTW, who is considered to be the release manager of this
>>>>>>>>> release?
>>>>>>>>>>>>>> —
>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>>>>>>>>>>>> dsetrakyan@apache.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> I like the format proposed by Denis, very clear.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, I also do not understand why a user should change
>>> the
>>>>>>>>>> size
>>>>>>>>>>>> of
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> system cache. How would a user ever know what value to put
>>>>>>>>> there?
>>>>>>>>>>>> This
>>>>>>>>>>>>>>> value should be configured by Ignite automatically.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>>>>>>>>> dmagda@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> Igor,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let me suggest this format.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>>>>> Excessive memory usage by Ignite node process
>> (performance
>>>>>>>>> may
>>>>>>>>>>> drop)
>>>>>>>>>>>>>>>> [requested=29251MB, available=15942MB]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please tune the following settings:
>>>>>>>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize =
>> suggested
>>>>>>>>>> value]
>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested
>> value
>>>>>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for
>> {policy_name_1} =
>>>>>>>>>>> suggested
>>>>>>>>>>>>>>>> value]
>>>>>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for
>> {policy_name_2} =
>>>>>>>>>>> suggested
>>>>>>>>>>>>>>>> value]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>>>>> [DefaultMemoryPolicySize = value]
>>>>>>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>>>>>>> SystemCacheInitialSize = value
>>>>>>>>>>>>>>>> SystemCacheMaxSize = value
>>>>>>>>>>>>>>>> Java Heap Init Size = value
>>>>>>>>>>>>>>>> Java Heap Max Size = value
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>>>>>>>>> value
>>>>>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Records in […] are optional. If custom memory policy is
>> not
>>>>>>>>> set
>>>>>>>>>> or
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> default memory policy is overridden the output will miss
>>>>>>>>> some of
>>>>>>>>>>> the
>>>>>>>>>>>>>> rows.
>>>>>>>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>>>>>>>>>> parameter
>>>>>>>>>>>> was
>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>> explicitly by user code. Otherwise, the platform should
>> be
>>>>>>>>> wise
>>>>>>>>>>>> enough
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> instantiate it properly depending on the host memory
>> usage.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> —
>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>>>>>>>>>>>> gvvinblade@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> The message without logging layout:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Not enough memory for current process [required=29251MB,
>>>>>>>>>>>>>>>> available=15942MB].
>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize
>> and
>>>>>>>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>>>>>>>>> memory
>>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>>>> for each node.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think it's make sense to describe what these numbers
>> are
>>>>>>>>>>> consist
>>>>>>>>>>>>> of.
>>>>>>>>>>>>>>>>> We simple say which parameters have an impact on how
>> much
>>>>>>>>>> memory
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> instance needs and their (parameters) actual values.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also we notice that more than one Ignite instance are
>> ran
>>> on
>>>>>>>>>> the
>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> workstation and it also consumes memory.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>>>>>>>>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after
>>> the
>>>>>>>>>>> text?
>>>>>>>>>>>>> Are
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> really going to print this whole thing out?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This message appears on topology change in case the
>>>>>>>>> available
>>>>>>>>>>>>> memory
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>> An example of current impl:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not
>>> enough
>>>>>>>>>>> memory
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> current process [required=29251MB,
>> available=15942MB].
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>> MemoryConfiguration.
>>>>>>>>>>>>>>>>>>> defaultMemoryPolicySize
>>>>>>>>>>>>>>>>>>>> to decrease memory allocated for each node.
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> Current
>>>>>>>>>>> settings:
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>> DefaultMemoryPolicySize=
>>>>>>>>>>>>>>>>>>>> 12753MB
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>>>>>>>>> ignite
>>>>>>>>>>>>>> instanses
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>> the server require: 12853MB
>>>>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
>>> ][disco-event-worker-#29%
>>>>>>>>>>>> internal.
>>>>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>>>>>>>>>>> skozlov@gridgain.com
>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
>>>>>>>>> suggest to
>>>>>>>>>>>>> reduce
>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>> memory
>>>>>>>>>>>> available
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty
>> and
>>>>>>>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to
>> decrease
>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> each node.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Also, can we calculate what the memory size
>> allocated
>>>>>>>>> for
>>>>>>>>>>> each
>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>> be? In that case we should suggest it.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>>>>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to
>> 40
>>>>>>>>> MB.
>>>>>>>>>>> Use
>>>>>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize
>> property
>>> to
>>>>>>>>>>> change
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> setting.
>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>>>>>>>> memory
>>>>>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap
>> partition
>>>>>>>>>> usage.
>>>>>>>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
>>>>>>>>> MemoryPolicyConfiguration
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> settings.
>>>>>>>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>>>>>>>>> available=15942MB]
>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>>>>>>>>> production
>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> deployment consistency reasons)
>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>>>>>>>>>>>>>>>> ['ignite-sys-cache']]
>>>>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>>>>>>>>> te/incubator-ignite/libs/
>>>>>>>>>>>>> licenses
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>>>>>>>>>>>> yzhdanov@apache.org
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>>>>> Agree, let's release new version including
>> tickets
>>>>>>>>>>> mentioned
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> Ivan.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --Yakov
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Sergey Kozlov
>>>>>>>>>>>>>>>>>>>>> GridGain Systems
>>>>>>>>>>>>>>>>>>>>> www.gridgain.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Igor Sapego <is...@gridgain.com>.
Cherry-picked from master. Changed "Fix version" in ticket.
Best Regards,
Igor
On Tue, Sep 5, 2017 at 4:15 PM, Anton Vinogradov <av...@gridgain.com>
wrote:
> Igor,
>
> That's an important fix.
> Please merge it to ignite-2.2.
> Don't forget to change fix version.
>
> On Tue, Sep 5, 2017 at 3:42 PM, Igor Sapego <is...@apache.org> wrote:
>
> > Igniters,
> >
> > There is one more fix I would propose to include into 2.2: [1]. It has
> > been fixed a while ago and it is critical for C++ users that would like
> > to make use of persistence store, as currently they need to have
> > separate Java node just to activate cluster.
> >
> > What do you think?
> >
> > [1] - https://issues.apache.org/jira/browse/IGNITE-5771
> >
> > Best Regards,
> > Igor
> >
> > On Mon, Sep 4, 2017 at 6:54 PM, Anton Vinogradov <
> avinogradov@gridgain.com
> > >
> > wrote:
> >
> > > Igniters,
> > >
> > > All fixes related to release 2.2 merged.
> > >
> > > But,
> > > we still waiting for changes related to new release process (
> > > https://issues.apache.org/jira/browse/IGNITE-5249).
> > > Beta version of "new release process" will be used to release 2.2,
> final
> > > changes will be merged to 2.3.
> > >
> > >
> > >
> > > On Fri, Sep 1, 2017 at 1:58 PM, Dmitry Pavlov <dp...@gmail.com>
> > > wrote:
> > >
> > > > Hi, at least for this release I suggest to keep versioning as is and
> > use
> > > > 2.2
> > > >
> > > > Without solved issue with http://apache-ignite-
> > > > developers.2346864.n4.nabble.com/usage-analytics-td19504.html Ignite
> > > will
> > > > not remind users about latest available ignite version. Versioning
> > > > structure change will confuse users for now. But keeping 2.2 will
> allow
> > > > Ignite users to take newest version with urgent fixes.
> > > >
> > > > I think is it important to achieve users to update. But without
> > > > notification server is it complex.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > пт, 1 сент. 2017 г. в 4:50, 李玉珏@163 <18...@163.com>:
> > > >
> > > >> Since it's an urgent bugfix version, why can't the version number be
> > > >> defined as 2.1.1 or 2.1.4?
> > > >> After all, functionality has not increased and documents do not
> > require
> > > >> big changes.
> > > >> If the version number is 2.1 series, there is no need to publish a
> new
> > > >> version of the document.
> > > >>
> > > >>
> > > >> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
> > > >> > Igniters,
> > > >> >
> > > >> > Seems 2.2 is a urgent bugfix release, so it should be based on
> 2.1,
> > > >> > In this case all other issues with fixVersion = 2.2 should be
> moved
> > to
> > > >> 2.3.
> > > >> >
> > > >> > Currently, I see 835 issues with fixVersion = 2.2
> > > >> >
> > > >> > Seems we should have only 4 issues with fixVersion = 2.2:
> > > >> > - Change default max memory size from 80% to 20% -
> > > >> > https://issues.apache.org/jira/browse/IGNITE-6182
> > > >> > - Detecting low memory on ignite node startup -
> > > >> > https://issues.apache.org/jira/browse/IGNITE-6003
> > > >> > - Backport of improvements to checkpoint algorithm -
> > > >> > https://issues.apache.org/jira/browse/IGNITE-????
> > > >> > - ML profile is missing in 2.1 binary release -
> > > >> > https://issues.apache.org/jira/browse/IGNITE-6193
> > > >> >
> > > >> > Please correct me in case I've missed something.
> > > >> >
> > > >> > Ivan,
> > > >> >> Let's include to the release tickets with optimizations of
> > > >> checkpointing
> > > >> > algorithm:
> > > >> >> https://issues.apache.org/jira/browse/IGNITE-6178
> > > >> >> https://issues.apache.org/jira/browse/IGNITE-6033
> > > >> >> https://issues.apache.org/jira/browse/IGNITE-5961
> > > >> >> This will help users who are experiencing problems with slow
> > > >> checkpoints.
> > > >> >> Also, let's include https://issues.apache.org/
> > > jira/browse/IGNITE-6183
> > > >> to
> > > >> > soften "Ignite node crashed in the middle of checkpoint" message
> as
> > > per
> > > >> > discussion <
> > > >> > http://apache-ignite-developers.2346864.n4.nabble.
> > > >> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
> > > >> false-td20473.html
> > > >> >> .
> > > >> > All of these issues should have fixVersion 2.3.
> > > >> > Please create backport issue and link it to all necessary issues.
> > > >> >
> > > >> > P.s. I've created branch ignite-2.2, currently it equals to
> > > ignite-2.1.
> > > >> >
> > > >> > On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
> > > >> avinogradov@gridgain.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Denis,
> > > >> >>
> > > >> >>> BTW, who is considered to be the release manager of this
> release?
> > > >> >> I'll do it.
> > > >> >>
> > > >> >> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
> > > >> gvvinblade@gmail.com>
> > > >> >> wrote:
> > > >> >>
> > > >> >>> Ok, the check happens at the node start time or on NODE_JOIN
> event
> > > >> >>>
> > > >> >>> in general it looks like:
> > > >> >>>
> > > >> >>> 1) calculate expected used memory = heap max + system cache max
> +
> > > all
> > > >> >>> custom policies max + default policy size and put it into a node
> > > >> attribute
> > > >> >>>
> > > >> >>> 2) get total physycal memory, calculate expected safe to be used
> > > >> memory
> > > >> >>> amount (leave 4 gb min or 20% of available memory for OS)
> > > >> >>>
> > > >> >>> 3) if expected used memory + expected used memory of other nodes
> > on
> > > >> the
> > > >> >>> host > than safe to be used memory amount, start calculating
> > > >> suggestions
> > > >> >>>
> > > >> >>> 4) Each ignite instance needs at least 512mb heap + 40mb system
> > > cache
> > > >> +
> > > >> >>> 100mb default polycy, if available memory is less we cannot
> > suggest
> > > >> >>> anything reasonable, print warning, stop calculation.
> > > >> >>>
> > > >> >>> 5) check heap size (shouldn't exceed 30% of available memory
> > > >> (total_memory
> > > >> >>> - reserved for OS memory) * 30% for all JVMs, if it exeedes,
> > suggest
> > > >> just
> > > >> >>> calculated value or 512MB minimal)
> > > >> >>>
> > > >> >>> 6) check if system cache size changed, suggest default value if
> > it's
> > > >> so
> > > >> >>>
> > > >> >>> 7) in case 100 mb * policies count < available memory, suggest
> > using
> > > >> >>> default policy with max size equals to remaining memory
> > (available -
> > > >> heap
> > > >> >>> -
> > > >> >>> system cache)
> > > >> >>>
> > > >> >>> 8) calculate new size for each memory policy ( it's user defined
> > > size
> > > >> *
> > > >> >>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> > > >> >>> remaining memory, devided by nodes number on the host or 100 mb
> > > >> minimal)
> > > >> >>>
> > > >> >>> 9) print suggestions
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > > >:
> > > >> >>>
> > > >> >>>> Igor, can you please describe the algorithm with all the
> > > thresholds?
> > > >> >>>>
> > > >> >>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
> > > >> gvvinblade@gmail.com
> > > >> >>>>
> > > >> >>>> wrote:
> > > >> >>>>
> > > >> >>>>> The suggestion here is based on initial settings, and it's so
> > > >> because
> > > >> >>>> there
> > > >> >>>>> is no other nodes on the host in the example.
> > > >> >>>>>
> > > >> >>>>> The algorithm tries to preserve the original ratio of memory
> > > >> policies
> > > >> >>>>> keeping numbers reasonable (for example after some thresshold
> it
> > > >> will
> > > >> >>>>> suggest not to use several memory policies if there is not
> > enough
> > > of
> > > >> >>>> memory
> > > >> >>>>> for all of them) and taking into consideration nodes count on
> > the
> > > >> >>> host,
> > > >> >>>>> each jvm heap, needed memory for OS, etc
> > > >> >>>>>
> > > >> >>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
> > > dsetrakyan@apache.org
> > > >> >:
> > > >> >>>>>
> > > >> >>>>>> Looks good, but why in the example provided are we suggesting
> > > 8GB?
> > > >> 2
> > > >> >>>>> nodes
> > > >> >>>>>> with 8GB will completely exhaust the available memory. I
> would
> > > >> >>> suggest
> > > >> >>>> 6
> > > >> >>>>> or
> > > >> >>>>>> 7GB.
> > > >> >>>>>>
> > > >> >>>>>> Also, why 100MB for default policy. Anything under 1GB seems
> > too
> > > >> >>> small.
> > > >> >>>>>> Can you please comment?
> > > >> >>>>>>
> > > >> >>>>>> D.
> > > >> >>>>>>
> > > >> >>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> > > >> >>>> gvvinblade@gmail.com>
> > > >> >>>>>> wrote:
> > > >> >>>>>>
> > > >> >>>>>>> One more example of possible warning:
> > > >> >>>>>>>
> > > >> >>>>>>> -----------------------------------------------------
> > > >> >>>>>>> Excessive memory usage by Ignite node process (performance
> may
> > > >> >>> drop)
> > > >> >>>>>>> [requested=44613MB, available=15942MB].
> > > >> >>>>>>>
> > > >> >>>>>>> Please tune the folowing settings as suggested:
> > > >> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > > >> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > > >> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > > >> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > > >> >>>>>>>
> > > >> >>>>>>> Current settings:
> > > >> >>>>>>> Java Heap maxSize: 3543MB
> > > >> >>>>>>> Java Heap initSize: 250MB
> > > >> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > > >> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc:
> 40960MB
> > > >> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > > >> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > > >> >>>>>>> The overall expected memory usage by all Ignite nodes on
> > the
> > > >> >>> host:
> > > >> >>>>>>> 44613MB
> > > >> >>>>>>> -----------------------------------------------------
> > > >> >>>>>>>
> > > >> >>>>>>> Your thoughts?
> > > >> >>>>>>>
> > > >> >>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> > > >> >>>>>>>
> > > >> >>>>>>>> Guys,
> > > >> >>>>>>>>
> > > >> >>>>>>>> ML lib profile is missing in 2.1 release! That must be
> fixed
> > > and
> > > >> >>>>> rolled
> > > >> >>>>>>>> out in this emergency release:
> > > >> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > >> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
> > > >> >>>>>>>>
> > > >> >>>>>>>> Oleg, Yuri, please step in and handle the issue.
> > > >> >>>>>>>>
> > > >> >>>>>>>> BTW, who is considered to be the release manager of this
> > > >> >>> release?
> > > >> >>>>>>>> —
> > > >> >>>>>>>> Denis
> > > >> >>>>>>>>
> > > >> >>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > > >> >>>>>> dsetrakyan@apache.org>
> > > >> >>>>>>>> wrote:
> > > >> >>>>>>>>> I like the format proposed by Denis, very clear.
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> However, I also do not understand why a user should change
> > the
> > > >> >>>> size
> > > >> >>>>>> of
> > > >> >>>>>>>> some
> > > >> >>>>>>>>> system cache. How would a user ever know what value to put
> > > >> >>> there?
> > > >> >>>>>> This
> > > >> >>>>>>>>> value should be configured by Ignite automatically.
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> D.
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> > > >> >>> dmagda@apache.org>
> > > >> >>>>>>> wrote:
> > > >> >>>>>>>>>> Igor,
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Let me suggest this format.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> ---------------------------------------------
> > > >> >>>>>>>>>> Excessive memory usage by Ignite node process
> (performance
> > > >> >>> may
> > > >> >>>>> drop)
> > > >> >>>>>>>>>> [requested=29251MB, available=15942MB]
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Please tune the following settings:
> > > >> >>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize =
> suggested
> > > >> >>>> value]
> > > >> >>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested
> value
> > > >> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for
> {policy_name_1} =
> > > >> >>>>> suggested
> > > >> >>>>>>>>>> value]
> > > >> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for
> {policy_name_2} =
> > > >> >>>>> suggested
> > > >> >>>>>>>>>> value]
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Current settings:
> > > >> >>>>>>>>>> [DefaultMemoryPolicySize = value]
> > > >> >>>>>>>>>> [{policy_name_1} size = value]
> > > >> >>>>>>>>>> [{policy_name_1} size = value]
> > > >> >>>>>>>>>> SystemCacheInitialSize = value
> > > >> >>>>>>>>>> SystemCacheMaxSize = value
> > > >> >>>>>>>>>> Java Heap Init Size = value
> > > >> >>>>>>>>>> Java Heap Max Size = value
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
> > > >> >>> value
> > > >> >>>>>>>>>> -------------------------------------------
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Records in […] are optional. If custom memory policy is
> not
> > > >> >>> set
> > > >> >>>> or
> > > >> >>>>>> the
> > > >> >>>>>>>>>> default memory policy is overridden the output will miss
> > > >> >>> some of
> > > >> >>>>> the
> > > >> >>>>>>>> rows.
> > > >> >>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
> > > >> >>>> parameter
> > > >> >>>>>> was
> > > >> >>>>>>>> set
> > > >> >>>>>>>>>> explicitly by user code. Otherwise, the platform should
> be
> > > >> >>> wise
> > > >> >>>>>> enough
> > > >> >>>>>>>> to
> > > >> >>>>>>>>>> instantiate it properly depending on the host memory
> usage.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> —
> > > >> >>>>>>>>>> Denis
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> > > >> >>>>>> gvvinblade@gmail.com>
> > > >> >>>>>>>>>> wrote:
> > > >> >>>>>>>>>>> The message without logging layout:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Not enough memory for current process [required=29251MB,
> > > >> >>>>>>>>>> available=15942MB].
> > > >> >>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize
> and
> > > >> >>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> > > >> >>> memory
> > > >> >>>>>>>> allocated
> > > >> >>>>>>>>>>> for each node.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Current settings:
> > > >> >>>>>>>>>>> HeapInit=250MB
> > > >> >>>>>>>>>>> HeapMax=3543MB
> > > >> >>>>>>>>>>> DefaultMemoryPolicySize=12753MB
> > > >> >>>>>>>>>>> SystemCacheInitialSize=40MB
> > > >> >>>>>>>>>>> SystemCacheMaxSize=100MB
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Other ignite instanses on the server require: 12853MB
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> I think it's make sense to describe what these numbers
> are
> > > >> >>>>> consist
> > > >> >>>>>>> of.
> > > >> >>>>>>>>>>> We simple say which parameters have an impact on how
> much
> > > >> >>>> memory
> > > >> >>>>>> the
> > > >> >>>>>>>>>>> instance needs and their (parameters) actual values.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> Also we notice that more than one Ignite instance are
> ran
> > on
> > > >> >>>> the
> > > >> >>>>>>> server
> > > >> >>>>>>>>>> or
> > > >> >>>>>>>>>>> workstation and it also consumes memory.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > > >> >>>>>>>>>>> dsetrakyan@apache.org> написал:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after
> > the
> > > >> >>>>> text?
> > > >> >>>>>>> Are
> > > >> >>>>>>>> we
> > > >> >>>>>>>>>>>> really going to print this whole thing out?
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> > > >> >>>>>>>> gvvinblade@gmail.com
> > > >> >>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> This message appears on topology change in case the
> > > >> >>> available
> > > >> >>>>>>> memory
> > > >> >>>>>>>> is
> > > >> >>>>>>>>>>>>> exceeded
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> > > >> >>>>>> gvvinblade@gmail.com
> > > >> >>>>>>>> :
> > > >> >>>>>>>>>>>>>> An example of current impl:
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not
> > enough
> > > >> >>>>> memory
> > > >> >>>>>>> for
> > > >> >>>>>>>>>>>>>> current process [required=29251MB,
> available=15942MB].
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> > > >> >>> change
> > > >> >>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
> > > >> >>>>> MemoryConfiguration.
> > > >> >>>>>>>>>>>>> defaultMemoryPolicySize
> > > >> >>>>>>>>>>>>>> to decrease memory allocated for each node.
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> Current
> > > >> >>>>> settings:
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>> HeapInit=250MB
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>> HeapMax=3543MB
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>>>>>>>>> DefaultMemoryPolicySize=
> > > >> >>>>>>>>>>>>>> 12753MB
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>>>>>>>>> SystemCacheMaxSize=100MB
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> > > >> >>> ignite
> > > >> >>>>>>>> instanses
> > > >> >>>>>>>>>>>> on
> > > >> >>>>>>>>>>>>>> the server require: 12853MB
> > > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> > ][disco-event-worker-#29%
> > > >> >>>>>> internal.
> > > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> > > >> >>>>> skozlov@gridgain.com
> > > >> >>>>>>> :
> > > >> >>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
> > > >> >>> suggest to
> > > >> >>>>>>> reduce
> > > >> >>>>>>>>>>>> both
> > > >> >>>>>>>>>>>>>>> options
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > > >> >>>>>>>>>>>>> dsetrakyan@apache.org
> > > >> >>>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> > memory
> > > >> >>>>>> available
> > > >> >>>>>>>> for
> > > >> >>>>>>>>>>>>> OS.
> > > >> >>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty
> and
> > > >> >>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to
> decrease
> > > >> >>>> memory
> > > >> >>>>>>>>>>>>> allocated
> > > >> >>>>>>>>>>>>>>> for
> > > >> >>>>>>>>>>>>>>>>> each node.
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> Also, can we calculate what the memory size
> allocated
> > > >> >>> for
> > > >> >>>>> each
> > > >> >>>>>>>> node
> > > >> >>>>>>>>>>>>>>> should
> > > >> >>>>>>>>>>>>>>>> be? In that case we should suggest it.
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> D.
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > > >> >>>>>>>>>>>>> gvvinblade@gmail.com
> > > >> >>>>>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>> What do you, guys think about next warning?
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > >> >>>>> GridHomePat
> > > >> >>>>>>>>>>>>>>> hSelfTest0]
> > > >> >>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to
> 40
> > > >> >>> MB.
> > > >> >>>>> Use
> > > >> >>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize
> property
> > to
> > > >> >>>>> change
> > > >> >>>>>>> the
> > > >> >>>>>>>>>>>>>>>> setting.
> > > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > >> >>>>> GridHomePat
> > > >> >>>>>>>>>>>>>>> hSelfTest0]
> > > >> >>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> > > >> >>> memory
> > > >> >>>>>>>>>>>> available
> > > >> >>>>>>>>>>>>>>> for
> > > >> >>>>>>>>>>>>>>>>> OS.
> > > >> >>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap
> partition
> > > >> >>>> usage.
> > > >> >>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
> > > >> >>> MemoryPolicyConfiguration
> > > >> >>>> to
> > > >> >>>>>>>>>>>> change
> > > >> >>>>>>>>>>>>>>> the
> > > >> >>>>>>>>>>>>>>>>> settings.
> > > >> >>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
> > > >> >>> available=15942MB]
> > > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > >> >>>>> GridHomePat
> > > >> >>>>>>>>>>>>>>> hSelfTest0]
> > > >> >>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
> > > >> >>> production
> > > >> >>>> for
> > > >> >>>>>>>>>>>>>>> performance
> > > >> >>>>>>>>>>>>>>>> and
> > > >> >>>>>>>>>>>>>>>>> deployment consistency reasons)
> > > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > >> >>>>> GridHomePat
> > > >> >>>>>>>>>>>>>>> hSelfTest0]
> > > >> >>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> > > >> >>>>>>>>>>>>> ['ignite-sys-cache']]
> > > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > >> >>>>> GridHomePat
> > > >> >>>>>>>>>>>>>>> hSelfTest0]
> > > >> >>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
> > > >> >>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
> > > >> >>> te/incubator-ignite/libs/
> > > >> >>>>>>> licenses
> > > >> >>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> > > >> >>>>>> yzhdanov@apache.org
> > > >> >>>>>>>> :
> > > >> >>>>>>>>>>>>>>>>>> Agree, let's release new version including
> tickets
> > > >> >>>>> mentioned
> > > >> >>>>>>> by
> > > >> >>>>>>>>>>>>>>> Denis
> > > >> >>>>>>>>>>>>>>>> and
> > > >> >>>>>>>>>>>>>>>>>> Ivan.
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>>>> --Yakov
> > > >> >>>>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>> --
> > > >> >>>>>>>>>>>>>>> Sergey Kozlov
> > > >> >>>>>>>>>>>>>>> GridGain Systems
> > > >> >>>>>>>>>>>>>>> www.gridgain.com
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>
> > > >> >>
> > > >>
> > > >>
> > > >>
> > >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Anton Vinogradov <av...@gridgain.com>.
Igor,
That's an important fix.
Please merge it to ignite-2.2.
Don't forget to change fix version.
On Tue, Sep 5, 2017 at 3:42 PM, Igor Sapego <is...@apache.org> wrote:
> Igniters,
>
> There is one more fix I would propose to include into 2.2: [1]. It has
> been fixed a while ago and it is critical for C++ users that would like
> to make use of persistence store, as currently they need to have
> separate Java node just to activate cluster.
>
> What do you think?
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-5771
>
> Best Regards,
> Igor
>
> On Mon, Sep 4, 2017 at 6:54 PM, Anton Vinogradov <avinogradov@gridgain.com
> >
> wrote:
>
> > Igniters,
> >
> > All fixes related to release 2.2 merged.
> >
> > But,
> > we still waiting for changes related to new release process (
> > https://issues.apache.org/jira/browse/IGNITE-5249).
> > Beta version of "new release process" will be used to release 2.2, final
> > changes will be merged to 2.3.
> >
> >
> >
> > On Fri, Sep 1, 2017 at 1:58 PM, Dmitry Pavlov <dp...@gmail.com>
> > wrote:
> >
> > > Hi, at least for this release I suggest to keep versioning as is and
> use
> > > 2.2
> > >
> > > Without solved issue with http://apache-ignite-
> > > developers.2346864.n4.nabble.com/usage-analytics-td19504.html Ignite
> > will
> > > not remind users about latest available ignite version. Versioning
> > > structure change will confuse users for now. But keeping 2.2 will allow
> > > Ignite users to take newest version with urgent fixes.
> > >
> > > I think is it important to achieve users to update. But without
> > > notification server is it complex.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > пт, 1 сент. 2017 г. в 4:50, 李玉珏@163 <18...@163.com>:
> > >
> > >> Since it's an urgent bugfix version, why can't the version number be
> > >> defined as 2.1.1 or 2.1.4?
> > >> After all, functionality has not increased and documents do not
> require
> > >> big changes.
> > >> If the version number is 2.1 series, there is no need to publish a new
> > >> version of the document.
> > >>
> > >>
> > >> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
> > >> > Igniters,
> > >> >
> > >> > Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> > >> > In this case all other issues with fixVersion = 2.2 should be moved
> to
> > >> 2.3.
> > >> >
> > >> > Currently, I see 835 issues with fixVersion = 2.2
> > >> >
> > >> > Seems we should have only 4 issues with fixVersion = 2.2:
> > >> > - Change default max memory size from 80% to 20% -
> > >> > https://issues.apache.org/jira/browse/IGNITE-6182
> > >> > - Detecting low memory on ignite node startup -
> > >> > https://issues.apache.org/jira/browse/IGNITE-6003
> > >> > - Backport of improvements to checkpoint algorithm -
> > >> > https://issues.apache.org/jira/browse/IGNITE-????
> > >> > - ML profile is missing in 2.1 binary release -
> > >> > https://issues.apache.org/jira/browse/IGNITE-6193
> > >> >
> > >> > Please correct me in case I've missed something.
> > >> >
> > >> > Ivan,
> > >> >> Let's include to the release tickets with optimizations of
> > >> checkpointing
> > >> > algorithm:
> > >> >> https://issues.apache.org/jira/browse/IGNITE-6178
> > >> >> https://issues.apache.org/jira/browse/IGNITE-6033
> > >> >> https://issues.apache.org/jira/browse/IGNITE-5961
> > >> >> This will help users who are experiencing problems with slow
> > >> checkpoints.
> > >> >> Also, let's include https://issues.apache.org/
> > jira/browse/IGNITE-6183
> > >> to
> > >> > soften "Ignite node crashed in the middle of checkpoint" message as
> > per
> > >> > discussion <
> > >> > http://apache-ignite-developers.2346864.n4.nabble.
> > >> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
> > >> false-td20473.html
> > >> >> .
> > >> > All of these issues should have fixVersion 2.3.
> > >> > Please create backport issue and link it to all necessary issues.
> > >> >
> > >> > P.s. I've created branch ignite-2.2, currently it equals to
> > ignite-2.1.
> > >> >
> > >> > On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
> > >> avinogradov@gridgain.com>
> > >> > wrote:
> > >> >
> > >> >> Denis,
> > >> >>
> > >> >>> BTW, who is considered to be the release manager of this release?
> > >> >> I'll do it.
> > >> >>
> > >> >> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
> > >> gvvinblade@gmail.com>
> > >> >> wrote:
> > >> >>
> > >> >>> Ok, the check happens at the node start time or on NODE_JOIN event
> > >> >>>
> > >> >>> in general it looks like:
> > >> >>>
> > >> >>> 1) calculate expected used memory = heap max + system cache max +
> > all
> > >> >>> custom policies max + default policy size and put it into a node
> > >> attribute
> > >> >>>
> > >> >>> 2) get total physycal memory, calculate expected safe to be used
> > >> memory
> > >> >>> amount (leave 4 gb min or 20% of available memory for OS)
> > >> >>>
> > >> >>> 3) if expected used memory + expected used memory of other nodes
> on
> > >> the
> > >> >>> host > than safe to be used memory amount, start calculating
> > >> suggestions
> > >> >>>
> > >> >>> 4) Each ignite instance needs at least 512mb heap + 40mb system
> > cache
> > >> +
> > >> >>> 100mb default polycy, if available memory is less we cannot
> suggest
> > >> >>> anything reasonable, print warning, stop calculation.
> > >> >>>
> > >> >>> 5) check heap size (shouldn't exceed 30% of available memory
> > >> (total_memory
> > >> >>> - reserved for OS memory) * 30% for all JVMs, if it exeedes,
> suggest
> > >> just
> > >> >>> calculated value or 512MB minimal)
> > >> >>>
> > >> >>> 6) check if system cache size changed, suggest default value if
> it's
> > >> so
> > >> >>>
> > >> >>> 7) in case 100 mb * policies count < available memory, suggest
> using
> > >> >>> default policy with max size equals to remaining memory
> (available -
> > >> heap
> > >> >>> -
> > >> >>> system cache)
> > >> >>>
> > >> >>> 8) calculate new size for each memory policy ( it's user defined
> > size
> > >> *
> > >> >>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> > >> >>> remaining memory, devided by nodes number on the host or 100 mb
> > >> minimal)
> > >> >>>
> > >> >>> 9) print suggestions
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >:
> > >> >>>
> > >> >>>> Igor, can you please describe the algorithm with all the
> > thresholds?
> > >> >>>>
> > >> >>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
> > >> gvvinblade@gmail.com
> > >> >>>>
> > >> >>>> wrote:
> > >> >>>>
> > >> >>>>> The suggestion here is based on initial settings, and it's so
> > >> because
> > >> >>>> there
> > >> >>>>> is no other nodes on the host in the example.
> > >> >>>>>
> > >> >>>>> The algorithm tries to preserve the original ratio of memory
> > >> policies
> > >> >>>>> keeping numbers reasonable (for example after some thresshold it
> > >> will
> > >> >>>>> suggest not to use several memory policies if there is not
> enough
> > of
> > >> >>>> memory
> > >> >>>>> for all of them) and taking into consideration nodes count on
> the
> > >> >>> host,
> > >> >>>>> each jvm heap, needed memory for OS, etc
> > >> >>>>>
> > >> >>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > >> >:
> > >> >>>>>
> > >> >>>>>> Looks good, but why in the example provided are we suggesting
> > 8GB?
> > >> 2
> > >> >>>>> nodes
> > >> >>>>>> with 8GB will completely exhaust the available memory. I would
> > >> >>> suggest
> > >> >>>> 6
> > >> >>>>> or
> > >> >>>>>> 7GB.
> > >> >>>>>>
> > >> >>>>>> Also, why 100MB for default policy. Anything under 1GB seems
> too
> > >> >>> small.
> > >> >>>>>> Can you please comment?
> > >> >>>>>>
> > >> >>>>>> D.
> > >> >>>>>>
> > >> >>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> > >> >>>> gvvinblade@gmail.com>
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>>> One more example of possible warning:
> > >> >>>>>>>
> > >> >>>>>>> -----------------------------------------------------
> > >> >>>>>>> Excessive memory usage by Ignite node process (performance may
> > >> >>> drop)
> > >> >>>>>>> [requested=44613MB, available=15942MB].
> > >> >>>>>>>
> > >> >>>>>>> Please tune the folowing settings as suggested:
> > >> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > >> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > >> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > >> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > >> >>>>>>>
> > >> >>>>>>> Current settings:
> > >> >>>>>>> Java Heap maxSize: 3543MB
> > >> >>>>>>> Java Heap initSize: 250MB
> > >> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > >> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > >> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > >> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > >> >>>>>>> The overall expected memory usage by all Ignite nodes on
> the
> > >> >>> host:
> > >> >>>>>>> 44613MB
> > >> >>>>>>> -----------------------------------------------------
> > >> >>>>>>>
> > >> >>>>>>> Your thoughts?
> > >> >>>>>>>
> > >> >>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> > >> >>>>>>>
> > >> >>>>>>>> Guys,
> > >> >>>>>>>>
> > >> >>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed
> > and
> > >> >>>>> rolled
> > >> >>>>>>>> out in this emergency release:
> > >> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
> > >> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
> > >> >>>>>>>>
> > >> >>>>>>>> Oleg, Yuri, please step in and handle the issue.
> > >> >>>>>>>>
> > >> >>>>>>>> BTW, who is considered to be the release manager of this
> > >> >>> release?
> > >> >>>>>>>> —
> > >> >>>>>>>> Denis
> > >> >>>>>>>>
> > >> >>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > >> >>>>>> dsetrakyan@apache.org>
> > >> >>>>>>>> wrote:
> > >> >>>>>>>>> I like the format proposed by Denis, very clear.
> > >> >>>>>>>>>
> > >> >>>>>>>>> However, I also do not understand why a user should change
> the
> > >> >>>> size
> > >> >>>>>> of
> > >> >>>>>>>> some
> > >> >>>>>>>>> system cache. How would a user ever know what value to put
> > >> >>> there?
> > >> >>>>>> This
> > >> >>>>>>>>> value should be configured by Ignite automatically.
> > >> >>>>>>>>>
> > >> >>>>>>>>> D.
> > >> >>>>>>>>>
> > >> >>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> > >> >>> dmagda@apache.org>
> > >> >>>>>>> wrote:
> > >> >>>>>>>>>> Igor,
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Let me suggest this format.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> ---------------------------------------------
> > >> >>>>>>>>>> Excessive memory usage by Ignite node process (performance
> > >> >>> may
> > >> >>>>> drop)
> > >> >>>>>>>>>> [requested=29251MB, available=15942MB]
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Please tune the following settings:
> > >> >>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
> > >> >>>> value]
> > >> >>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
> > >> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> > >> >>>>> suggested
> > >> >>>>>>>>>> value]
> > >> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> > >> >>>>> suggested
> > >> >>>>>>>>>> value]
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Current settings:
> > >> >>>>>>>>>> [DefaultMemoryPolicySize = value]
> > >> >>>>>>>>>> [{policy_name_1} size = value]
> > >> >>>>>>>>>> [{policy_name_1} size = value]
> > >> >>>>>>>>>> SystemCacheInitialSize = value
> > >> >>>>>>>>>> SystemCacheMaxSize = value
> > >> >>>>>>>>>> Java Heap Init Size = value
> > >> >>>>>>>>>> Java Heap Max Size = value
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
> > >> >>> value
> > >> >>>>>>>>>> -------------------------------------------
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Records in […] are optional. If custom memory policy is not
> > >> >>> set
> > >> >>>> or
> > >> >>>>>> the
> > >> >>>>>>>>>> default memory policy is overridden the output will miss
> > >> >>> some of
> > >> >>>>> the
> > >> >>>>>>>> rows.
> > >> >>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
> > >> >>>> parameter
> > >> >>>>>> was
> > >> >>>>>>>> set
> > >> >>>>>>>>>> explicitly by user code. Otherwise, the platform should be
> > >> >>> wise
> > >> >>>>>> enough
> > >> >>>>>>>> to
> > >> >>>>>>>>>> instantiate it properly depending on the host memory usage.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> —
> > >> >>>>>>>>>> Denis
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> > >> >>>>>> gvvinblade@gmail.com>
> > >> >>>>>>>>>> wrote:
> > >> >>>>>>>>>>> The message without logging layout:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Not enough memory for current process [required=29251MB,
> > >> >>>>>>>>>> available=15942MB].
> > >> >>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
> > >> >>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> > >> >>> memory
> > >> >>>>>>>> allocated
> > >> >>>>>>>>>>> for each node.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Current settings:
> > >> >>>>>>>>>>> HeapInit=250MB
> > >> >>>>>>>>>>> HeapMax=3543MB
> > >> >>>>>>>>>>> DefaultMemoryPolicySize=12753MB
> > >> >>>>>>>>>>> SystemCacheInitialSize=40MB
> > >> >>>>>>>>>>> SystemCacheMaxSize=100MB
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Other ignite instanses on the server require: 12853MB
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I think it's make sense to describe what these numbers are
> > >> >>>>> consist
> > >> >>>>>>> of.
> > >> >>>>>>>>>>> We simple say which parameters have an impact on how much
> > >> >>>> memory
> > >> >>>>>> the
> > >> >>>>>>>>>>> instance needs and their (parameters) actual values.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Also we notice that more than one Ignite instance are ran
> on
> > >> >>>> the
> > >> >>>>>>> server
> > >> >>>>>>>>>> or
> > >> >>>>>>>>>>> workstation and it also consumes memory.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > >> >>>>>>>>>>> dsetrakyan@apache.org> написал:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after
> the
> > >> >>>>> text?
> > >> >>>>>>> Are
> > >> >>>>>>>> we
> > >> >>>>>>>>>>>> really going to print this whole thing out?
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> > >> >>>>>>>> gvvinblade@gmail.com
> > >> >>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>> This message appears on topology change in case the
> > >> >>> available
> > >> >>>>>>> memory
> > >> >>>>>>>> is
> > >> >>>>>>>>>>>>> exceeded
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> > >> >>>>>> gvvinblade@gmail.com
> > >> >>>>>>>> :
> > >> >>>>>>>>>>>>>> An example of current impl:
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not
> enough
> > >> >>>>> memory
> > >> >>>>>>> for
> > >> >>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> > >> >>> change
> > >> >>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
> > >> >>>>> MemoryConfiguration.
> > >> >>>>>>>>>>>>> defaultMemoryPolicySize
> > >> >>>>>>>>>>>>>> to decrease memory allocated for each node.
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> > >> >>>>> settings:
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>> HeapInit=250MB
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>> HeapMax=3543MB
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>>>>>>>>> DefaultMemoryPolicySize=
> > >> >>>>>>>>>>>>>> 12753MB
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>>>>>>>>> SystemCacheMaxSize=100MB
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> > >> >>> ignite
> > >> >>>>>>>> instanses
> > >> >>>>>>>>>>>> on
> > >> >>>>>>>>>>>>>> the server require: 12853MB
> > >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN
> ][disco-event-worker-#29%
> > >> >>>>>> internal.
> > >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> > >> >>>>> skozlov@gridgain.com
> > >> >>>>>>> :
> > >> >>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
> > >> >>> suggest to
> > >> >>>>>>> reduce
> > >> >>>>>>>>>>>> both
> > >> >>>>>>>>>>>>>>> options
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > >> >>>>>>>>>>>>> dsetrakyan@apache.org
> > >> >>>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> memory
> > >> >>>>>> available
> > >> >>>>>>>> for
> > >> >>>>>>>>>>>>> OS.
> > >> >>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> > >> >>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
> > >> >>>> memory
> > >> >>>>>>>>>>>>> allocated
> > >> >>>>>>>>>>>>>>> for
> > >> >>>>>>>>>>>>>>>>> each node.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
> > >> >>> for
> > >> >>>>> each
> > >> >>>>>>>> node
> > >> >>>>>>>>>>>>>>> should
> > >> >>>>>>>>>>>>>>>> be? In that case we should suggest it.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> D.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > >> >>>>>>>>>>>>> gvvinblade@gmail.com
> > >> >>>>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>> What do you, guys think about next warning?
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >> >>>>> GridHomePat
> > >> >>>>>>>>>>>>>>> hSelfTest0]
> > >> >>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
> > >> >>> MB.
> > >> >>>>> Use
> > >> >>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property
> to
> > >> >>>>> change
> > >> >>>>>>> the
> > >> >>>>>>>>>>>>>>>> setting.
> > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >> >>>>> GridHomePat
> > >> >>>>>>>>>>>>>>> hSelfTest0]
> > >> >>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> > >> >>> memory
> > >> >>>>>>>>>>>> available
> > >> >>>>>>>>>>>>>>> for
> > >> >>>>>>>>>>>>>>>>> OS.
> > >> >>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
> > >> >>>> usage.
> > >> >>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
> > >> >>> MemoryPolicyConfiguration
> > >> >>>> to
> > >> >>>>>>>>>>>> change
> > >> >>>>>>>>>>>>>>> the
> > >> >>>>>>>>>>>>>>>>> settings.
> > >> >>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
> > >> >>> available=15942MB]
> > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >> >>>>> GridHomePat
> > >> >>>>>>>>>>>>>>> hSelfTest0]
> > >> >>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
> > >> >>> production
> > >> >>>> for
> > >> >>>>>>>>>>>>>>> performance
> > >> >>>>>>>>>>>>>>>> and
> > >> >>>>>>>>>>>>>>>>> deployment consistency reasons)
> > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >> >>>>> GridHomePat
> > >> >>>>>>>>>>>>>>> hSelfTest0]
> > >> >>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> > >> >>>>>>>>>>>>> ['ignite-sys-cache']]
> > >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >> >>>>> GridHomePat
> > >> >>>>>>>>>>>>>>> hSelfTest0]
> > >> >>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
> > >> >>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
> > >> >>> te/incubator-ignite/libs/
> > >> >>>>>>> licenses
> > >> >>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> > >> >>>>>> yzhdanov@apache.org
> > >> >>>>>>>> :
> > >> >>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
> > >> >>>>> mentioned
> > >> >>>>>>> by
> > >> >>>>>>>>>>>>>>> Denis
> > >> >>>>>>>>>>>>>>>> and
> > >> >>>>>>>>>>>>>>>>>> Ivan.
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>>> --Yakov
> > >> >>>>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> --
> > >> >>>>>>>>>>>>>>> Sergey Kozlov
> > >> >>>>>>>>>>>>>>> GridGain Systems
> > >> >>>>>>>>>>>>>>> www.gridgain.com
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>
> > >> >>
> > >>
> > >>
> > >>
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Igor Sapego <is...@apache.org>.
Igniters,
There is one more fix I would propose to include into 2.2: [1]. It has
been fixed a while ago and it is critical for C++ users that would like
to make use of persistence store, as currently they need to have
separate Java node just to activate cluster.
What do you think?
[1] - https://issues.apache.org/jira/browse/IGNITE-5771
Best Regards,
Igor
On Mon, Sep 4, 2017 at 6:54 PM, Anton Vinogradov <av...@gridgain.com>
wrote:
> Igniters,
>
> All fixes related to release 2.2 merged.
>
> But,
> we still waiting for changes related to new release process (
> https://issues.apache.org/jira/browse/IGNITE-5249).
> Beta version of "new release process" will be used to release 2.2, final
> changes will be merged to 2.3.
>
>
>
> On Fri, Sep 1, 2017 at 1:58 PM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > Hi, at least for this release I suggest to keep versioning as is and use
> > 2.2
> >
> > Without solved issue with http://apache-ignite-
> > developers.2346864.n4.nabble.com/usage-analytics-td19504.html Ignite
> will
> > not remind users about latest available ignite version. Versioning
> > structure change will confuse users for now. But keeping 2.2 will allow
> > Ignite users to take newest version with urgent fixes.
> >
> > I think is it important to achieve users to update. But without
> > notification server is it complex.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пт, 1 сент. 2017 г. в 4:50, 李玉珏@163 <18...@163.com>:
> >
> >> Since it's an urgent bugfix version, why can't the version number be
> >> defined as 2.1.1 or 2.1.4?
> >> After all, functionality has not increased and documents do not require
> >> big changes.
> >> If the version number is 2.1 series, there is no need to publish a new
> >> version of the document.
> >>
> >>
> >> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
> >> > Igniters,
> >> >
> >> > Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> >> > In this case all other issues with fixVersion = 2.2 should be moved to
> >> 2.3.
> >> >
> >> > Currently, I see 835 issues with fixVersion = 2.2
> >> >
> >> > Seems we should have only 4 issues with fixVersion = 2.2:
> >> > - Change default max memory size from 80% to 20% -
> >> > https://issues.apache.org/jira/browse/IGNITE-6182
> >> > - Detecting low memory on ignite node startup -
> >> > https://issues.apache.org/jira/browse/IGNITE-6003
> >> > - Backport of improvements to checkpoint algorithm -
> >> > https://issues.apache.org/jira/browse/IGNITE-????
> >> > - ML profile is missing in 2.1 binary release -
> >> > https://issues.apache.org/jira/browse/IGNITE-6193
> >> >
> >> > Please correct me in case I've missed something.
> >> >
> >> > Ivan,
> >> >> Let's include to the release tickets with optimizations of
> >> checkpointing
> >> > algorithm:
> >> >> https://issues.apache.org/jira/browse/IGNITE-6178
> >> >> https://issues.apache.org/jira/browse/IGNITE-6033
> >> >> https://issues.apache.org/jira/browse/IGNITE-5961
> >> >> This will help users who are experiencing problems with slow
> >> checkpoints.
> >> >> Also, let's include https://issues.apache.org/
> jira/browse/IGNITE-6183
> >> to
> >> > soften "Ignite node crashed in the middle of checkpoint" message as
> per
> >> > discussion <
> >> > http://apache-ignite-developers.2346864.n4.nabble.
> >> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
> >> false-td20473.html
> >> >> .
> >> > All of these issues should have fixVersion 2.3.
> >> > Please create backport issue and link it to all necessary issues.
> >> >
> >> > P.s. I've created branch ignite-2.2, currently it equals to
> ignite-2.1.
> >> >
> >> > On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
> >> avinogradov@gridgain.com>
> >> > wrote:
> >> >
> >> >> Denis,
> >> >>
> >> >>> BTW, who is considered to be the release manager of this release?
> >> >> I'll do it.
> >> >>
> >> >> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
> >> gvvinblade@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Ok, the check happens at the node start time or on NODE_JOIN event
> >> >>>
> >> >>> in general it looks like:
> >> >>>
> >> >>> 1) calculate expected used memory = heap max + system cache max +
> all
> >> >>> custom policies max + default policy size and put it into a node
> >> attribute
> >> >>>
> >> >>> 2) get total physycal memory, calculate expected safe to be used
> >> memory
> >> >>> amount (leave 4 gb min or 20% of available memory for OS)
> >> >>>
> >> >>> 3) if expected used memory + expected used memory of other nodes on
> >> the
> >> >>> host > than safe to be used memory amount, start calculating
> >> suggestions
> >> >>>
> >> >>> 4) Each ignite instance needs at least 512mb heap + 40mb system
> cache
> >> +
> >> >>> 100mb default polycy, if available memory is less we cannot suggest
> >> >>> anything reasonable, print warning, stop calculation.
> >> >>>
> >> >>> 5) check heap size (shouldn't exceed 30% of available memory
> >> (total_memory
> >> >>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
> >> just
> >> >>> calculated value or 512MB minimal)
> >> >>>
> >> >>> 6) check if system cache size changed, suggest default value if it's
> >> so
> >> >>>
> >> >>> 7) in case 100 mb * policies count < available memory, suggest using
> >> >>> default policy with max size equals to remaining memory (available -
> >> heap
> >> >>> -
> >> >>> system cache)
> >> >>>
> >> >>> 8) calculate new size for each memory policy ( it's user defined
> size
> >> *
> >> >>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> >> >>> remaining memory, devided by nodes number on the host or 100 mb
> >> minimal)
> >> >>>
> >> >>> 9) print suggestions
> >> >>>
> >> >>>
> >> >>>
> >> >>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
> >:
> >> >>>
> >> >>>> Igor, can you please describe the algorithm with all the
> thresholds?
> >> >>>>
> >> >>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
> >> gvvinblade@gmail.com
> >> >>>>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> The suggestion here is based on initial settings, and it's so
> >> because
> >> >>>> there
> >> >>>>> is no other nodes on the host in the example.
> >> >>>>>
> >> >>>>> The algorithm tries to preserve the original ratio of memory
> >> policies
> >> >>>>> keeping numbers reasonable (for example after some thresshold it
> >> will
> >> >>>>> suggest not to use several memory policies if there is not enough
> of
> >> >>>> memory
> >> >>>>> for all of them) and taking into consideration nodes count on the
> >> >>> host,
> >> >>>>> each jvm heap, needed memory for OS, etc
> >> >>>>>
> >> >>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
> dsetrakyan@apache.org
> >> >:
> >> >>>>>
> >> >>>>>> Looks good, but why in the example provided are we suggesting
> 8GB?
> >> 2
> >> >>>>> nodes
> >> >>>>>> with 8GB will completely exhaust the available memory. I would
> >> >>> suggest
> >> >>>> 6
> >> >>>>> or
> >> >>>>>> 7GB.
> >> >>>>>>
> >> >>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
> >> >>> small.
> >> >>>>>> Can you please comment?
> >> >>>>>>
> >> >>>>>> D.
> >> >>>>>>
> >> >>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> >> >>>> gvvinblade@gmail.com>
> >> >>>>>> wrote:
> >> >>>>>>
> >> >>>>>>> One more example of possible warning:
> >> >>>>>>>
> >> >>>>>>> -----------------------------------------------------
> >> >>>>>>> Excessive memory usage by Ignite node process (performance may
> >> >>> drop)
> >> >>>>>>> [requested=44613MB, available=15942MB].
> >> >>>>>>>
> >> >>>>>>> Please tune the folowing settings as suggested:
> >> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> >> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> >> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> >> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> >> >>>>>>>
> >> >>>>>>> Current settings:
> >> >>>>>>> Java Heap maxSize: 3543MB
> >> >>>>>>> Java Heap initSize: 250MB
> >> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> >> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> >> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> >> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> >> >>>>>>> The overall expected memory usage by all Ignite nodes on the
> >> >>> host:
> >> >>>>>>> 44613MB
> >> >>>>>>> -----------------------------------------------------
> >> >>>>>>>
> >> >>>>>>> Your thoughts?
> >> >>>>>>>
> >> >>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> >> >>>>>>>
> >> >>>>>>>> Guys,
> >> >>>>>>>>
> >> >>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed
> and
> >> >>>>> rolled
> >> >>>>>>>> out in this emergency release:
> >> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
> >> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
> >> >>>>>>>>
> >> >>>>>>>> Oleg, Yuri, please step in and handle the issue.
> >> >>>>>>>>
> >> >>>>>>>> BTW, who is considered to be the release manager of this
> >> >>> release?
> >> >>>>>>>> —
> >> >>>>>>>> Denis
> >> >>>>>>>>
> >> >>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> >> >>>>>> dsetrakyan@apache.org>
> >> >>>>>>>> wrote:
> >> >>>>>>>>> I like the format proposed by Denis, very clear.
> >> >>>>>>>>>
> >> >>>>>>>>> However, I also do not understand why a user should change the
> >> >>>> size
> >> >>>>>> of
> >> >>>>>>>> some
> >> >>>>>>>>> system cache. How would a user ever know what value to put
> >> >>> there?
> >> >>>>>> This
> >> >>>>>>>>> value should be configured by Ignite automatically.
> >> >>>>>>>>>
> >> >>>>>>>>> D.
> >> >>>>>>>>>
> >> >>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> >> >>> dmagda@apache.org>
> >> >>>>>>> wrote:
> >> >>>>>>>>>> Igor,
> >> >>>>>>>>>>
> >> >>>>>>>>>> Let me suggest this format.
> >> >>>>>>>>>>
> >> >>>>>>>>>> ---------------------------------------------
> >> >>>>>>>>>> Excessive memory usage by Ignite node process (performance
> >> >>> may
> >> >>>>> drop)
> >> >>>>>>>>>> [requested=29251MB, available=15942MB]
> >> >>>>>>>>>>
> >> >>>>>>>>>> Please tune the following settings:
> >> >>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
> >> >>>> value]
> >> >>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
> >> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> >> >>>>> suggested
> >> >>>>>>>>>> value]
> >> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> >> >>>>> suggested
> >> >>>>>>>>>> value]
> >> >>>>>>>>>>
> >> >>>>>>>>>> Current settings:
> >> >>>>>>>>>> [DefaultMemoryPolicySize = value]
> >> >>>>>>>>>> [{policy_name_1} size = value]
> >> >>>>>>>>>> [{policy_name_1} size = value]
> >> >>>>>>>>>> SystemCacheInitialSize = value
> >> >>>>>>>>>> SystemCacheMaxSize = value
> >> >>>>>>>>>> Java Heap Init Size = value
> >> >>>>>>>>>> Java Heap Max Size = value
> >> >>>>>>>>>>
> >> >>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
> >> >>> value
> >> >>>>>>>>>> -------------------------------------------
> >> >>>>>>>>>>
> >> >>>>>>>>>> Records in […] are optional. If custom memory policy is not
> >> >>> set
> >> >>>> or
> >> >>>>>> the
> >> >>>>>>>>>> default memory policy is overridden the output will miss
> >> >>> some of
> >> >>>>> the
> >> >>>>>>>> rows.
> >> >>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
> >> >>>> parameter
> >> >>>>>> was
> >> >>>>>>>> set
> >> >>>>>>>>>> explicitly by user code. Otherwise, the platform should be
> >> >>> wise
> >> >>>>>> enough
> >> >>>>>>>> to
> >> >>>>>>>>>> instantiate it properly depending on the host memory usage.
> >> >>>>>>>>>>
> >> >>>>>>>>>> —
> >> >>>>>>>>>> Denis
> >> >>>>>>>>>>
> >> >>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> >> >>>>>> gvvinblade@gmail.com>
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>>> The message without logging layout:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Not enough memory for current process [required=29251MB,
> >> >>>>>>>>>> available=15942MB].
> >> >>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
> >> >>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> >> >>> memory
> >> >>>>>>>> allocated
> >> >>>>>>>>>>> for each node.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Current settings:
> >> >>>>>>>>>>> HeapInit=250MB
> >> >>>>>>>>>>> HeapMax=3543MB
> >> >>>>>>>>>>> DefaultMemoryPolicySize=12753MB
> >> >>>>>>>>>>> SystemCacheInitialSize=40MB
> >> >>>>>>>>>>> SystemCacheMaxSize=100MB
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Other ignite instanses on the server require: 12853MB
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I think it's make sense to describe what these numbers are
> >> >>>>> consist
> >> >>>>>>> of.
> >> >>>>>>>>>>> We simple say which parameters have an impact on how much
> >> >>>> memory
> >> >>>>>> the
> >> >>>>>>>>>>> instance needs and their (parameters) actual values.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
> >> >>>> the
> >> >>>>>>> server
> >> >>>>>>>>>> or
> >> >>>>>>>>>>> workstation and it also consumes memory.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> >> >>>>>>>>>>> dsetrakyan@apache.org> написал:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
> >> >>>>> text?
> >> >>>>>>> Are
> >> >>>>>>>> we
> >> >>>>>>>>>>>> really going to print this whole thing out?
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> >> >>>>>>>> gvvinblade@gmail.com
> >> >>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>> This message appears on topology change in case the
> >> >>> available
> >> >>>>>>> memory
> >> >>>>>>>> is
> >> >>>>>>>>>>>>> exceeded
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> >> >>>>>> gvvinblade@gmail.com
> >> >>>>>>>> :
> >> >>>>>>>>>>>>>> An example of current impl:
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
> >> >>>>> memory
> >> >>>>>>> for
> >> >>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> >> >>> change
> >> >>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
> >> >>>>> MemoryConfiguration.
> >> >>>>>>>>>>>>> defaultMemoryPolicySize
> >> >>>>>>>>>>>>>> to decrease memory allocated for each node.
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> >> >>>>> settings:
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>> HeapInit=250MB
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>> HeapMax=3543MB
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>>>>>>>>> DefaultMemoryPolicySize=
> >> >>>>>>>>>>>>>> 12753MB
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>>>>>>>>> SystemCacheMaxSize=100MB
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> >> >>> ignite
> >> >>>>>>>> instanses
> >> >>>>>>>>>>>> on
> >> >>>>>>>>>>>>>> the server require: 12853MB
> >> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >> >>>>>> internal.
> >> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> >> >>>>> skozlov@gridgain.com
> >> >>>>>>> :
> >> >>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
> >> >>> suggest to
> >> >>>>>>> reduce
> >> >>>>>>>>>>>> both
> >> >>>>>>>>>>>>>>> options
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> >> >>>>>>>>>>>>> dsetrakyan@apache.org
> >> >>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
> >> >>>>>> available
> >> >>>>>>>> for
> >> >>>>>>>>>>>>> OS.
> >> >>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> >> >>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
> >> >>>> memory
> >> >>>>>>>>>>>>> allocated
> >> >>>>>>>>>>>>>>> for
> >> >>>>>>>>>>>>>>>>> each node.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
> >> >>> for
> >> >>>>> each
> >> >>>>>>>> node
> >> >>>>>>>>>>>>>>> should
> >> >>>>>>>>>>>>>>>> be? In that case we should suggest it.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> D.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> >> >>>>>>>>>>>>> gvvinblade@gmail.com
> >> >>>>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> What do you, guys think about next warning?
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
> >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >> >>>>> GridHomePat
> >> >>>>>>>>>>>>>>> hSelfTest0]
> >> >>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
> >> >>> MB.
> >> >>>>> Use
> >> >>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
> >> >>>>> change
> >> >>>>>>> the
> >> >>>>>>>>>>>>>>>> setting.
> >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
> >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >> >>>>> GridHomePat
> >> >>>>>>>>>>>>>>> hSelfTest0]
> >> >>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> >> >>> memory
> >> >>>>>>>>>>>> available
> >> >>>>>>>>>>>>>>> for
> >> >>>>>>>>>>>>>>>>> OS.
> >> >>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
> >> >>>> usage.
> >> >>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
> >> >>> MemoryPolicyConfiguration
> >> >>>> to
> >> >>>>>>>>>>>> change
> >> >>>>>>>>>>>>>>> the
> >> >>>>>>>>>>>>>>>>> settings.
> >> >>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
> >> >>> available=15942MB]
> >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
> >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >> >>>>> GridHomePat
> >> >>>>>>>>>>>>>>> hSelfTest0]
> >> >>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
> >> >>> production
> >> >>>> for
> >> >>>>>>>>>>>>>>> performance
> >> >>>>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>> deployment consistency reasons)
> >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
> >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >> >>>>> GridHomePat
> >> >>>>>>>>>>>>>>> hSelfTest0]
> >> >>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> >> >>>>>>>>>>>>> ['ignite-sys-cache']]
> >> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
> >> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >> >>>>> GridHomePat
> >> >>>>>>>>>>>>>>> hSelfTest0]
> >> >>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
> >> >>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
> >> >>> te/incubator-ignite/libs/
> >> >>>>>>> licenses
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> >> >>>>>> yzhdanov@apache.org
> >> >>>>>>>> :
> >> >>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
> >> >>>>> mentioned
> >> >>>>>>> by
> >> >>>>>>>>>>>>>>> Denis
> >> >>>>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>> Ivan.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> --Yakov
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> --
> >> >>>>>>>>>>>>>>> Sergey Kozlov
> >> >>>>>>>>>>>>>>> GridGain Systems
> >> >>>>>>>>>>>>>>> www.gridgain.com
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>
> >> >>
> >>
> >>
> >>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Anton Vinogradov <av...@gridgain.com>.
Igniters,
All fixes related to release 2.2 merged.
But,
we still waiting for changes related to new release process (
https://issues.apache.org/jira/browse/IGNITE-5249).
Beta version of "new release process" will be used to release 2.2, final
changes will be merged to 2.3.
On Fri, Sep 1, 2017 at 1:58 PM, Dmitry Pavlov <dp...@gmail.com> wrote:
> Hi, at least for this release I suggest to keep versioning as is and use
> 2.2
>
> Without solved issue with http://apache-ignite-
> developers.2346864.n4.nabble.com/usage-analytics-td19504.html Ignite will
> not remind users about latest available ignite version. Versioning
> structure change will confuse users for now. But keeping 2.2 will allow
> Ignite users to take newest version with urgent fixes.
>
> I think is it important to achieve users to update. But without
> notification server is it complex.
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 1 сент. 2017 г. в 4:50, 李玉珏@163 <18...@163.com>:
>
>> Since it's an urgent bugfix version, why can't the version number be
>> defined as 2.1.1 or 2.1.4?
>> After all, functionality has not increased and documents do not require
>> big changes.
>> If the version number is 2.1 series, there is no need to publish a new
>> version of the document.
>>
>>
>> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
>> > Igniters,
>> >
>> > Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
>> > In this case all other issues with fixVersion = 2.2 should be moved to
>> 2.3.
>> >
>> > Currently, I see 835 issues with fixVersion = 2.2
>> >
>> > Seems we should have only 4 issues with fixVersion = 2.2:
>> > - Change default max memory size from 80% to 20% -
>> > https://issues.apache.org/jira/browse/IGNITE-6182
>> > - Detecting low memory on ignite node startup -
>> > https://issues.apache.org/jira/browse/IGNITE-6003
>> > - Backport of improvements to checkpoint algorithm -
>> > https://issues.apache.org/jira/browse/IGNITE-????
>> > - ML profile is missing in 2.1 binary release -
>> > https://issues.apache.org/jira/browse/IGNITE-6193
>> >
>> > Please correct me in case I've missed something.
>> >
>> > Ivan,
>> >> Let's include to the release tickets with optimizations of
>> checkpointing
>> > algorithm:
>> >> https://issues.apache.org/jira/browse/IGNITE-6178
>> >> https://issues.apache.org/jira/browse/IGNITE-6033
>> >> https://issues.apache.org/jira/browse/IGNITE-5961
>> >> This will help users who are experiencing problems with slow
>> checkpoints.
>> >> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183
>> to
>> > soften "Ignite node crashed in the middle of checkpoint" message as per
>> > discussion <
>> > http://apache-ignite-developers.2346864.n4.nabble.
>> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
>> false-td20473.html
>> >> .
>> > All of these issues should have fixVersion 2.3.
>> > Please create backport issue and link it to all necessary issues.
>> >
>> > P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
>> >
>> > On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
>> avinogradov@gridgain.com>
>> > wrote:
>> >
>> >> Denis,
>> >>
>> >>> BTW, who is considered to be the release manager of this release?
>> >> I'll do it.
>> >>
>> >> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
>> gvvinblade@gmail.com>
>> >> wrote:
>> >>
>> >>> Ok, the check happens at the node start time or on NODE_JOIN event
>> >>>
>> >>> in general it looks like:
>> >>>
>> >>> 1) calculate expected used memory = heap max + system cache max + all
>> >>> custom policies max + default policy size and put it into a node
>> attribute
>> >>>
>> >>> 2) get total physycal memory, calculate expected safe to be used
>> memory
>> >>> amount (leave 4 gb min or 20% of available memory for OS)
>> >>>
>> >>> 3) if expected used memory + expected used memory of other nodes on
>> the
>> >>> host > than safe to be used memory amount, start calculating
>> suggestions
>> >>>
>> >>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache
>> +
>> >>> 100mb default polycy, if available memory is less we cannot suggest
>> >>> anything reasonable, print warning, stop calculation.
>> >>>
>> >>> 5) check heap size (shouldn't exceed 30% of available memory
>> (total_memory
>> >>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
>> just
>> >>> calculated value or 512MB minimal)
>> >>>
>> >>> 6) check if system cache size changed, suggest default value if it's
>> so
>> >>>
>> >>> 7) in case 100 mb * policies count < available memory, suggest using
>> >>> default policy with max size equals to remaining memory (available -
>> heap
>> >>> -
>> >>> system cache)
>> >>>
>> >>> 8) calculate new size for each memory policy ( it's user defined size
>> *
>> >>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>> >>> remaining memory, devided by nodes number on the host or 100 mb
>> minimal)
>> >>>
>> >>> 9) print suggestions
>> >>>
>> >>>
>> >>>
>> >>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>> >>>
>> >>>> Igor, can you please describe the algorithm with all the thresholds?
>> >>>>
>> >>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
>> gvvinblade@gmail.com
>> >>>>
>> >>>> wrote:
>> >>>>
>> >>>>> The suggestion here is based on initial settings, and it's so
>> because
>> >>>> there
>> >>>>> is no other nodes on the host in the example.
>> >>>>>
>> >>>>> The algorithm tries to preserve the original ratio of memory
>> policies
>> >>>>> keeping numbers reasonable (for example after some thresshold it
>> will
>> >>>>> suggest not to use several memory policies if there is not enough of
>> >>>> memory
>> >>>>> for all of them) and taking into consideration nodes count on the
>> >>> host,
>> >>>>> each jvm heap, needed memory for OS, etc
>> >>>>>
>> >>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
>> >:
>> >>>>>
>> >>>>>> Looks good, but why in the example provided are we suggesting 8GB?
>> 2
>> >>>>> nodes
>> >>>>>> with 8GB will completely exhaust the available memory. I would
>> >>> suggest
>> >>>> 6
>> >>>>> or
>> >>>>>> 7GB.
>> >>>>>>
>> >>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
>> >>> small.
>> >>>>>> Can you please comment?
>> >>>>>>
>> >>>>>> D.
>> >>>>>>
>> >>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>> >>>> gvvinblade@gmail.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> One more example of possible warning:
>> >>>>>>>
>> >>>>>>> -----------------------------------------------------
>> >>>>>>> Excessive memory usage by Ignite node process (performance may
>> >>> drop)
>> >>>>>>> [requested=44613MB, available=15942MB].
>> >>>>>>>
>> >>>>>>> Please tune the folowing settings as suggested:
>> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>> >>>>>>>
>> >>>>>>> Current settings:
>> >>>>>>> Java Heap maxSize: 3543MB
>> >>>>>>> Java Heap initSize: 250MB
>> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>> >>>>>>> The overall expected memory usage by all Ignite nodes on the
>> >>> host:
>> >>>>>>> 44613MB
>> >>>>>>> -----------------------------------------------------
>> >>>>>>>
>> >>>>>>> Your thoughts?
>> >>>>>>>
>> >>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>> >>>>>>>
>> >>>>>>>> Guys,
>> >>>>>>>>
>> >>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
>> >>>>> rolled
>> >>>>>>>> out in this emergency release:
>> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>> >>>>>>>>
>> >>>>>>>> Oleg, Yuri, please step in and handle the issue.
>> >>>>>>>>
>> >>>>>>>> BTW, who is considered to be the release manager of this
>> >>> release?
>> >>>>>>>> —
>> >>>>>>>> Denis
>> >>>>>>>>
>> >>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>> >>>>>> dsetrakyan@apache.org>
>> >>>>>>>> wrote:
>> >>>>>>>>> I like the format proposed by Denis, very clear.
>> >>>>>>>>>
>> >>>>>>>>> However, I also do not understand why a user should change the
>> >>>> size
>> >>>>>> of
>> >>>>>>>> some
>> >>>>>>>>> system cache. How would a user ever know what value to put
>> >>> there?
>> >>>>>> This
>> >>>>>>>>> value should be configured by Ignite automatically.
>> >>>>>>>>>
>> >>>>>>>>> D.
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>> >>> dmagda@apache.org>
>> >>>>>>> wrote:
>> >>>>>>>>>> Igor,
>> >>>>>>>>>>
>> >>>>>>>>>> Let me suggest this format.
>> >>>>>>>>>>
>> >>>>>>>>>> ---------------------------------------------
>> >>>>>>>>>> Excessive memory usage by Ignite node process (performance
>> >>> may
>> >>>>> drop)
>> >>>>>>>>>> [requested=29251MB, available=15942MB]
>> >>>>>>>>>>
>> >>>>>>>>>> Please tune the following settings:
>> >>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
>> >>>> value]
>> >>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
>> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
>> >>>>> suggested
>> >>>>>>>>>> value]
>> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
>> >>>>> suggested
>> >>>>>>>>>> value]
>> >>>>>>>>>>
>> >>>>>>>>>> Current settings:
>> >>>>>>>>>> [DefaultMemoryPolicySize = value]
>> >>>>>>>>>> [{policy_name_1} size = value]
>> >>>>>>>>>> [{policy_name_1} size = value]
>> >>>>>>>>>> SystemCacheInitialSize = value
>> >>>>>>>>>> SystemCacheMaxSize = value
>> >>>>>>>>>> Java Heap Init Size = value
>> >>>>>>>>>> Java Heap Max Size = value
>> >>>>>>>>>>
>> >>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>> >>> value
>> >>>>>>>>>> -------------------------------------------
>> >>>>>>>>>>
>> >>>>>>>>>> Records in […] are optional. If custom memory policy is not
>> >>> set
>> >>>> or
>> >>>>>> the
>> >>>>>>>>>> default memory policy is overridden the output will miss
>> >>> some of
>> >>>>> the
>> >>>>>>>> rows.
>> >>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>> >>>> parameter
>> >>>>>> was
>> >>>>>>>> set
>> >>>>>>>>>> explicitly by user code. Otherwise, the platform should be
>> >>> wise
>> >>>>>> enough
>> >>>>>>>> to
>> >>>>>>>>>> instantiate it properly depending on the host memory usage.
>> >>>>>>>>>>
>> >>>>>>>>>> —
>> >>>>>>>>>> Denis
>> >>>>>>>>>>
>> >>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>> >>>>>> gvvinblade@gmail.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>> The message without logging layout:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Not enough memory for current process [required=29251MB,
>> >>>>>>>>>> available=15942MB].
>> >>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
>> >>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>> >>> memory
>> >>>>>>>> allocated
>> >>>>>>>>>>> for each node.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Current settings:
>> >>>>>>>>>>> HeapInit=250MB
>> >>>>>>>>>>> HeapMax=3543MB
>> >>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>> >>>>>>>>>>> SystemCacheInitialSize=40MB
>> >>>>>>>>>>> SystemCacheMaxSize=100MB
>> >>>>>>>>>>>
>> >>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>> >>>>>>>>>>>
>> >>>>>>>>>>> I think it's make sense to describe what these numbers are
>> >>>>> consist
>> >>>>>>> of.
>> >>>>>>>>>>> We simple say which parameters have an impact on how much
>> >>>> memory
>> >>>>>> the
>> >>>>>>>>>>> instance needs and their (parameters) actual values.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
>> >>>> the
>> >>>>>>> server
>> >>>>>>>>>> or
>> >>>>>>>>>>> workstation and it also consumes memory.
>> >>>>>>>>>>>
>> >>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>> >>>>>>>>>>> dsetrakyan@apache.org> написал:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
>> >>>>> text?
>> >>>>>>> Are
>> >>>>>>>> we
>> >>>>>>>>>>>> really going to print this whole thing out?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>> >>>>>>>> gvvinblade@gmail.com
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> This message appears on topology change in case the
>> >>> available
>> >>>>>>> memory
>> >>>>>>>> is
>> >>>>>>>>>>>>> exceeded
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>> >>>>>> gvvinblade@gmail.com
>> >>>>>>>> :
>> >>>>>>>>>>>>>> An example of current impl:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
>> >>>>> memory
>> >>>>>>> for
>> >>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>> >>> change
>> >>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
>> >>>>> MemoryConfiguration.
>> >>>>>>>>>>>>> defaultMemoryPolicySize
>> >>>>>>>>>>>>>> to decrease memory allocated for each node.
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
>> >>>>> settings:
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>> HeapInit=250MB
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>> HeapMax=3543MB
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>>>>>>>>> DefaultMemoryPolicySize=
>> >>>>>>>>>>>>>> 12753MB
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>>>>>>>>> SystemCacheMaxSize=100MB
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>> >>> ignite
>> >>>>>>>> instanses
>> >>>>>>>>>>>> on
>> >>>>>>>>>>>>>> the server require: 12853MB
>> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> >>>>>> internal.
>> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>> >>>>> skozlov@gridgain.com
>> >>>>>>> :
>> >>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
>> >>> suggest to
>> >>>>>>> reduce
>> >>>>>>>>>>>> both
>> >>>>>>>>>>>>>>> options
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>> >>>>>>>>>>>>> dsetrakyan@apache.org
>> >>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
>> >>>>>> available
>> >>>>>>>> for
>> >>>>>>>>>>>>> OS.
>> >>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>> >>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
>> >>>> memory
>> >>>>>>>>>>>>> allocated
>> >>>>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>> each node.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
>> >>> for
>> >>>>> each
>> >>>>>>>> node
>> >>>>>>>>>>>>>>> should
>> >>>>>>>>>>>>>>>> be? In that case we should suggest it.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> D.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>> >>>>>>>>>>>>> gvvinblade@gmail.com
>> >>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> >>>>> GridHomePat
>> >>>>>>>>>>>>>>> hSelfTest0]
>> >>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
>> >>> MB.
>> >>>>> Use
>> >>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
>> >>>>> change
>> >>>>>>> the
>> >>>>>>>>>>>>>>>> setting.
>> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> >>>>> GridHomePat
>> >>>>>>>>>>>>>>> hSelfTest0]
>> >>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>> >>> memory
>> >>>>>>>>>>>> available
>> >>>>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>> OS.
>> >>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
>> >>>> usage.
>> >>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
>> >>> MemoryPolicyConfiguration
>> >>>> to
>> >>>>>>>>>>>> change
>> >>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>> settings.
>> >>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>> >>> available=15942MB]
>> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> >>>>> GridHomePat
>> >>>>>>>>>>>>>>> hSelfTest0]
>> >>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>> >>> production
>> >>>> for
>> >>>>>>>>>>>>>>> performance
>> >>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>> deployment consistency reasons)
>> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> >>>>> GridHomePat
>> >>>>>>>>>>>>>>> hSelfTest0]
>> >>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>> >>>>>>>>>>>>> ['ignite-sys-cache']]
>> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> >>>>> GridHomePat
>> >>>>>>>>>>>>>>> hSelfTest0]
>> >>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>> >>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>> >>> te/incubator-ignite/libs/
>> >>>>>>> licenses
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>> >>>>>> yzhdanov@apache.org
>> >>>>>>>> :
>> >>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
>> >>>>> mentioned
>> >>>>>>> by
>> >>>>>>>>>>>>>>> Denis
>> >>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>> Ivan.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> --Yakov
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>> Sergey Kozlov
>> >>>>>>>>>>>>>>> GridGain Systems
>> >>>>>>>>>>>>>>> www.gridgain.com
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>
>>
>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi, at least for this release I suggest to keep versioning as is and use 2.2
Without solved issue with
http://apache-ignite-developers.2346864.n4.nabble.com/usage-analytics-td19504.html
Ignite will not remind users about latest available ignite version.
Versioning structure change will confuse users for now. But keeping 2.2
will allow Ignite users to take newest version with urgent fixes.
I think is it important to achieve users to update. But without
notification server is it complex.
Sincerely,
Dmitriy Pavlov
пт, 1 сент. 2017 г. в 4:50, 李玉珏@163 <18...@163.com>:
> Since it's an urgent bugfix version, why can't the version number be
> defined as 2.1.1 or 2.1.4?
> After all, functionality has not increased and documents do not require
> big changes.
> If the version number is 2.1 series, there is no need to publish a new
> version of the document.
>
>
> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
> > Igniters,
> >
> > Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> > In this case all other issues with fixVersion = 2.2 should be moved to
> 2.3.
> >
> > Currently, I see 835 issues with fixVersion = 2.2
> >
> > Seems we should have only 4 issues with fixVersion = 2.2:
> > - Change default max memory size from 80% to 20% -
> > https://issues.apache.org/jira/browse/IGNITE-6182
> > - Detecting low memory on ignite node startup -
> > https://issues.apache.org/jira/browse/IGNITE-6003
> > - Backport of improvements to checkpoint algorithm -
> > https://issues.apache.org/jira/browse/IGNITE-????
> > - ML profile is missing in 2.1 binary release -
> > https://issues.apache.org/jira/browse/IGNITE-6193
> >
> > Please correct me in case I've missed something.
> >
> > Ivan,
> >> Let's include to the release tickets with optimizations of checkpointing
> > algorithm:
> >> https://issues.apache.org/jira/browse/IGNITE-6178
> >> https://issues.apache.org/jira/browse/IGNITE-6033
> >> https://issues.apache.org/jira/browse/IGNITE-5961
> >> This will help users who are experiencing problems with slow
> checkpoints.
> >> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183
> to
> > soften "Ignite node crashed in the middle of checkpoint" message as per
> > discussion <
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
> >> .
> > All of these issues should have fixVersion 2.3.
> > Please create backport issue and link it to all necessary issues.
> >
> > P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
> >
> > On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
> avinogradov@gridgain.com>
> > wrote:
> >
> >> Denis,
> >>
> >>> BTW, who is considered to be the release manager of this release?
> >> I'll do it.
> >>
> >> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gvvinblade@gmail.com
> >
> >> wrote:
> >>
> >>> Ok, the check happens at the node start time or on NODE_JOIN event
> >>>
> >>> in general it looks like:
> >>>
> >>> 1) calculate expected used memory = heap max + system cache max + all
> >>> custom policies max + default policy size and put it into a node
> attribute
> >>>
> >>> 2) get total physycal memory, calculate expected safe to be used memory
> >>> amount (leave 4 gb min or 20% of available memory for OS)
> >>>
> >>> 3) if expected used memory + expected used memory of other nodes on the
> >>> host > than safe to be used memory amount, start calculating
> suggestions
> >>>
> >>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
> >>> 100mb default polycy, if available memory is less we cannot suggest
> >>> anything reasonable, print warning, stop calculation.
> >>>
> >>> 5) check heap size (shouldn't exceed 30% of available memory
> (total_memory
> >>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
> just
> >>> calculated value or 512MB minimal)
> >>>
> >>> 6) check if system cache size changed, suggest default value if it's so
> >>>
> >>> 7) in case 100 mb * policies count < available memory, suggest using
> >>> default policy with max size equals to remaining memory (available -
> heap
> >>> -
> >>> system cache)
> >>>
> >>> 8) calculate new size for each memory policy ( it's user defined size *
> >>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> >>> remaining memory, devided by nodes number on the host or 100 mb
> minimal)
> >>>
> >>> 9) print suggestions
> >>>
> >>>
> >>>
> >>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> >>>
> >>>> Igor, can you please describe the algorithm with all the thresholds?
> >>>>
> >>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
> gvvinblade@gmail.com
> >>>>
> >>>> wrote:
> >>>>
> >>>>> The suggestion here is based on initial settings, and it's so because
> >>>> there
> >>>>> is no other nodes on the host in the example.
> >>>>>
> >>>>> The algorithm tries to preserve the original ratio of memory policies
> >>>>> keeping numbers reasonable (for example after some thresshold it will
> >>>>> suggest not to use several memory policies if there is not enough of
> >>>> memory
> >>>>> for all of them) and taking into consideration nodes count on the
> >>> host,
> >>>>> each jvm heap, needed memory for OS, etc
> >>>>>
> >>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
> >:
> >>>>>
> >>>>>> Looks good, but why in the example provided are we suggesting 8GB? 2
> >>>>> nodes
> >>>>>> with 8GB will completely exhaust the available memory. I would
> >>> suggest
> >>>> 6
> >>>>> or
> >>>>>> 7GB.
> >>>>>>
> >>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
> >>> small.
> >>>>>> Can you please comment?
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> >>>> gvvinblade@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> One more example of possible warning:
> >>>>>>>
> >>>>>>> -----------------------------------------------------
> >>>>>>> Excessive memory usage by Ignite node process (performance may
> >>> drop)
> >>>>>>> [requested=44613MB, available=15942MB].
> >>>>>>>
> >>>>>>> Please tune the folowing settings as suggested:
> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> >>>>>>>
> >>>>>>> Current settings:
> >>>>>>> Java Heap maxSize: 3543MB
> >>>>>>> Java Heap initSize: 250MB
> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> >>>>>>> The overall expected memory usage by all Ignite nodes on the
> >>> host:
> >>>>>>> 44613MB
> >>>>>>> -----------------------------------------------------
> >>>>>>>
> >>>>>>> Your thoughts?
> >>>>>>>
> >>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> >>>>>>>
> >>>>>>>> Guys,
> >>>>>>>>
> >>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
> >>>>> rolled
> >>>>>>>> out in this emergency release:
> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
> >>>>>>>>
> >>>>>>>> Oleg, Yuri, please step in and handle the issue.
> >>>>>>>>
> >>>>>>>> BTW, who is considered to be the release manager of this
> >>> release?
> >>>>>>>> —
> >>>>>>>> Denis
> >>>>>>>>
> >>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> >>>>>> dsetrakyan@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>> I like the format proposed by Denis, very clear.
> >>>>>>>>>
> >>>>>>>>> However, I also do not understand why a user should change the
> >>>> size
> >>>>>> of
> >>>>>>>> some
> >>>>>>>>> system cache. How would a user ever know what value to put
> >>> there?
> >>>>>> This
> >>>>>>>>> value should be configured by Ignite automatically.
> >>>>>>>>>
> >>>>>>>>> D.
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> >>> dmagda@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>> Igor,
> >>>>>>>>>>
> >>>>>>>>>> Let me suggest this format.
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------
> >>>>>>>>>> Excessive memory usage by Ignite node process (performance
> >>> may
> >>>>> drop)
> >>>>>>>>>> [requested=29251MB, available=15942MB]
> >>>>>>>>>>
> >>>>>>>>>> Please tune the following settings:
> >>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
> >>>> value]
> >>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> >>>>> suggested
> >>>>>>>>>> value]
> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> >>>>> suggested
> >>>>>>>>>> value]
> >>>>>>>>>>
> >>>>>>>>>> Current settings:
> >>>>>>>>>> [DefaultMemoryPolicySize = value]
> >>>>>>>>>> [{policy_name_1} size = value]
> >>>>>>>>>> [{policy_name_1} size = value]
> >>>>>>>>>> SystemCacheInitialSize = value
> >>>>>>>>>> SystemCacheMaxSize = value
> >>>>>>>>>> Java Heap Init Size = value
> >>>>>>>>>> Java Heap Max Size = value
> >>>>>>>>>>
> >>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
> >>> value
> >>>>>>>>>> -------------------------------------------
> >>>>>>>>>>
> >>>>>>>>>> Records in […] are optional. If custom memory policy is not
> >>> set
> >>>> or
> >>>>>> the
> >>>>>>>>>> default memory policy is overridden the output will miss
> >>> some of
> >>>>> the
> >>>>>>>> rows.
> >>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
> >>>> parameter
> >>>>>> was
> >>>>>>>> set
> >>>>>>>>>> explicitly by user code. Otherwise, the platform should be
> >>> wise
> >>>>>> enough
> >>>>>>>> to
> >>>>>>>>>> instantiate it properly depending on the host memory usage.
> >>>>>>>>>>
> >>>>>>>>>> —
> >>>>>>>>>> Denis
> >>>>>>>>>>
> >>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> >>>>>> gvvinblade@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>> The message without logging layout:
> >>>>>>>>>>>
> >>>>>>>>>>> Not enough memory for current process [required=29251MB,
> >>>>>>>>>> available=15942MB].
> >>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
> >>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> >>> memory
> >>>>>>>> allocated
> >>>>>>>>>>> for each node.
> >>>>>>>>>>>
> >>>>>>>>>>> Current settings:
> >>>>>>>>>>> HeapInit=250MB
> >>>>>>>>>>> HeapMax=3543MB
> >>>>>>>>>>> DefaultMemoryPolicySize=12753MB
> >>>>>>>>>>> SystemCacheInitialSize=40MB
> >>>>>>>>>>> SystemCacheMaxSize=100MB
> >>>>>>>>>>>
> >>>>>>>>>>> Other ignite instanses on the server require: 12853MB
> >>>>>>>>>>>
> >>>>>>>>>>> I think it's make sense to describe what these numbers are
> >>>>> consist
> >>>>>>> of.
> >>>>>>>>>>> We simple say which parameters have an impact on how much
> >>>> memory
> >>>>>> the
> >>>>>>>>>>> instance needs and their (parameters) actual values.
> >>>>>>>>>>>
> >>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
> >>>> the
> >>>>>>> server
> >>>>>>>>>> or
> >>>>>>>>>>> workstation and it also consumes memory.
> >>>>>>>>>>>
> >>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> >>>>>>>>>>> dsetrakyan@apache.org> написал:
> >>>>>>>>>>>
> >>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
> >>>>> text?
> >>>>>>> Are
> >>>>>>>> we
> >>>>>>>>>>>> really going to print this whole thing out?
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> >>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> This message appears on topology change in case the
> >>> available
> >>>>>>> memory
> >>>>>>>> is
> >>>>>>>>>>>>> exceeded
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> >>>>>> gvvinblade@gmail.com
> >>>>>>>> :
> >>>>>>>>>>>>>> An example of current impl:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
> >>>>> memory
> >>>>>>> for
> >>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> >>> change
> >>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
> >>>>> MemoryConfiguration.
> >>>>>>>>>>>>> defaultMemoryPolicySize
> >>>>>>>>>>>>>> to decrease memory allocated for each node.
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> >>>>> settings:
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>> HeapInit=250MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>> HeapMax=3543MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>> DefaultMemoryPolicySize=
> >>>>>>>>>>>>>> 12753MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>> SystemCacheMaxSize=100MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> >>> ignite
> >>>>>>>> instanses
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>> the server require: 12853MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> >>>>> skozlov@gridgain.com
> >>>>>>> :
> >>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
> >>> suggest to
> >>>>>>> reduce
> >>>>>>>>>>>> both
> >>>>>>>>>>>>>>> options
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>>> dsetrakyan@apache.org
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
> >>>>>> available
> >>>>>>>> for
> >>>>>>>>>>>>> OS.
> >>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> >>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
> >>>> memory
> >>>>>>>>>>>>> allocated
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> each node.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
> >>> for
> >>>>> each
> >>>>>>>> node
> >>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>> be? In that case we should suggest it.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> >>>>>>>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> What do you, guys think about next warning?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
> >>> MB.
> >>>>> Use
> >>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
> >>>>> change
> >>>>>>> the
> >>>>>>>>>>>>>>>> setting.
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> >>> memory
> >>>>>>>>>>>> available
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> OS.
> >>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
> >>>> usage.
> >>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
> >>> MemoryPolicyConfiguration
> >>>> to
> >>>>>>>>>>>> change
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> settings.
> >>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
> >>> available=15942MB]
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
> >>> production
> >>>> for
> >>>>>>>>>>>>>>> performance
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> deployment consistency reasons)
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> >>>>>>>>>>>>> ['ignite-sys-cache']]
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
> >>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
> >>> te/incubator-ignite/libs/
> >>>>>>> licenses
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> >>>>>> yzhdanov@apache.org
> >>>>>>>> :
> >>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
> >>>>> mentioned
> >>>>>>> by
> >>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> Ivan.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --Yakov
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Sergey Kozlov
> >>>>>>>>>>>>>>> GridGain Systems
> >>>>>>>>>>>>>>> www.gridgain.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>
>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Anton Vinogradov <av...@gridgain.com>.
Hi,
Reasonable question, but I propose to keep 2.2 as a release version for
this urgent release, since we never released X.Y.Z version before.
Let's discuss this at separate thread.
Could you please start it?
On Fri, Sep 1, 2017 at 4:49 AM, 李玉珏@163 <18...@163.com> wrote:
> Since it's an urgent bugfix version, why can't the version number be
> defined as 2.1.1 or 2.1.4?
> After all, functionality has not increased and documents do not require
> big changes.
> If the version number is 2.1 series, there is no need to publish a new
> version of the document.
>
>
>
> 在 2017/8/28 下午10:19, Anton Vinogradov 写道:
>
>> Igniters,
>>
>> Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
>> In this case all other issues with fixVersion = 2.2 should be moved to
>> 2.3.
>>
>> Currently, I see 835 issues with fixVersion = 2.2
>>
>> Seems we should have only 4 issues with fixVersion = 2.2:
>> - Change default max memory size from 80% to 20% -
>> https://issues.apache.org/jira/browse/IGNITE-6182
>> - Detecting low memory on ignite node startup -
>> https://issues.apache.org/jira/browse/IGNITE-6003
>> - Backport of improvements to checkpoint algorithm -
>> https://issues.apache.org/jira/browse/IGNITE-????
>> - ML profile is missing in 2.1 binary release -
>> https://issues.apache.org/jira/browse/IGNITE-6193
>>
>> Please correct me in case I've missed something.
>>
>> Ivan,
>>
>>> Let's include to the release tickets with optimizations of checkpointing
>>>
>> algorithm:
>>
>>> https://issues.apache.org/jira/browse/IGNITE-6178
>>> https://issues.apache.org/jira/browse/IGNITE-6033
>>> https://issues.apache.org/jira/browse/IGNITE-5961
>>> This will help users who are experiencing problems with slow checkpoints.
>>> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to
>>>
>> soften "Ignite node crashed in the middle of checkpoint" message as per
>> discussion <
>> http://apache-ignite-developers.2346864.n4.nabble.com/
>> Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>>
>>> .
>>>
>> All of these issues should have fixVersion 2.3.
>> Please create backport issue and link it to all necessary issues.
>>
>> P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
>>
>> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
>> avinogradov@gridgain.com>
>> wrote:
>>
>> Denis,
>>>
>>> BTW, who is considered to be the release manager of this release?
>>>>
>>> I'll do it.
>>>
>>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gv...@gmail.com>
>>> wrote:
>>>
>>> Ok, the check happens at the node start time or on NODE_JOIN event
>>>>
>>>> in general it looks like:
>>>>
>>>> 1) calculate expected used memory = heap max + system cache max + all
>>>> custom policies max + default policy size and put it into a node
>>>> attribute
>>>>
>>>> 2) get total physycal memory, calculate expected safe to be used memory
>>>> amount (leave 4 gb min or 20% of available memory for OS)
>>>>
>>>> 3) if expected used memory + expected used memory of other nodes on the
>>>> host > than safe to be used memory amount, start calculating suggestions
>>>>
>>>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
>>>> 100mb default polycy, if available memory is less we cannot suggest
>>>> anything reasonable, print warning, stop calculation.
>>>>
>>>> 5) check heap size (shouldn't exceed 30% of available memory
>>>> (total_memory
>>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
>>>> just
>>>> calculated value or 512MB minimal)
>>>>
>>>> 6) check if system cache size changed, suggest default value if it's so
>>>>
>>>> 7) in case 100 mb * policies count < available memory, suggest using
>>>> default policy with max size equals to remaining memory (available -
>>>> heap
>>>> -
>>>> system cache)
>>>>
>>>> 8) calculate new size for each memory policy ( it's user defined size *
>>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>>> remaining memory, devided by nodes number on the host or 100 mb minimal)
>>>>
>>>> 9) print suggestions
>>>>
>>>>
>>>>
>>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>>
>>>> Igor, can you please describe the algorithm with all the thresholds?
>>>>>
>>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
>>>>> gvvinblade@gmail.com
>>>>>
>>>>> wrote:
>>>>>
>>>>> The suggestion here is based on initial settings, and it's so because
>>>>>>
>>>>> there
>>>>>
>>>>>> is no other nodes on the host in the example.
>>>>>>
>>>>>> The algorithm tries to preserve the original ratio of memory policies
>>>>>> keeping numbers reasonable (for example after some thresshold it will
>>>>>> suggest not to use several memory policies if there is not enough of
>>>>>>
>>>>> memory
>>>>>
>>>>>> for all of them) and taking into consideration nodes count on the
>>>>>>
>>>>> host,
>>>>
>>>>> each jvm heap, needed memory for OS, etc
>>>>>>
>>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>>>>
>>>>>> Looks good, but why in the example provided are we suggesting 8GB? 2
>>>>>>>
>>>>>> nodes
>>>>>>
>>>>>>> with 8GB will completely exhaust the available memory. I would
>>>>>>>
>>>>>> suggest
>>>>
>>>>> 6
>>>>>
>>>>>> or
>>>>>>
>>>>>>> 7GB.
>>>>>>>
>>>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
>>>>>>>
>>>>>> small.
>>>>
>>>>> Can you please comment?
>>>>>>>
>>>>>>> D.
>>>>>>>
>>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>>>>>>>
>>>>>> gvvinblade@gmail.com>
>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> One more example of possible warning:
>>>>>>>>
>>>>>>>> -----------------------------------------------------
>>>>>>>> Excessive memory usage by Ignite node process (performance may
>>>>>>>>
>>>>>>> drop)
>>>>
>>>>> [requested=44613MB, available=15942MB].
>>>>>>>>
>>>>>>>> Please tune the folowing settings as suggested:
>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>>>>>>>>
>>>>>>>> Current settings:
>>>>>>>> Java Heap maxSize: 3543MB
>>>>>>>> Java Heap initSize: 250MB
>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>>>>>>>> The overall expected memory usage by all Ignite nodes on the
>>>>>>>>
>>>>>>> host:
>>>>
>>>>> 44613MB
>>>>>>>> -----------------------------------------------------
>>>>>>>>
>>>>>>>> Your thoughts?
>>>>>>>>
>>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>>>>>>>>
>>>>>>>> Guys,
>>>>>>>>>
>>>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
>>>>>>>>>
>>>>>>>> rolled
>>>>>>
>>>>>>> out in this emergency release:
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>>>>>>>>>
>>>>>>>>> Oleg, Yuri, please step in and handle the issue.
>>>>>>>>>
>>>>>>>>> BTW, who is considered to be the release manager of this
>>>>>>>>>
>>>>>>>> release?
>>>>
>>>>> —
>>>>>>>>> Denis
>>>>>>>>>
>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>>>>>>>>>>
>>>>>>>>> dsetrakyan@apache.org>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I like the format proposed by Denis, very clear.
>>>>>>>>>>
>>>>>>>>>> However, I also do not understand why a user should change the
>>>>>>>>>>
>>>>>>>>> size
>>>>>
>>>>>> of
>>>>>>>
>>>>>>>> some
>>>>>>>>>
>>>>>>>>>> system cache. How would a user ever know what value to put
>>>>>>>>>>
>>>>>>>>> there?
>>>>
>>>>> This
>>>>>>>
>>>>>>>> value should be configured by Ignite automatically.
>>>>>>>>>>
>>>>>>>>>> D.
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>>>>>>>>>>
>>>>>>>>> dmagda@apache.org>
>>>>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> Igor,
>>>>>>>>>>>
>>>>>>>>>>> Let me suggest this format.
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>> Excessive memory usage by Ignite node process (performance
>>>>>>>>>>>
>>>>>>>>>> may
>>>>
>>>>> drop)
>>>>>>
>>>>>>> [requested=29251MB, available=15942MB]
>>>>>>>>>>>
>>>>>>>>>>> Please tune the following settings:
>>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
>>>>>>>>>>>
>>>>>>>>>> value]
>>>>>
>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
>>>>>>>>>>>
>>>>>>>>>> suggested
>>>>>>
>>>>>>> value]
>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
>>>>>>>>>>>
>>>>>>>>>> suggested
>>>>>>
>>>>>>> value]
>>>>>>>>>>>
>>>>>>>>>>> Current settings:
>>>>>>>>>>> [DefaultMemoryPolicySize = value]
>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>> SystemCacheInitialSize = value
>>>>>>>>>>> SystemCacheMaxSize = value
>>>>>>>>>>> Java Heap Init Size = value
>>>>>>>>>>> Java Heap Max Size = value
>>>>>>>>>>>
>>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>>>>>>>>>>>
>>>>>>>>>> value
>>>>
>>>>> -------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> Records in […] are optional. If custom memory policy is not
>>>>>>>>>>>
>>>>>>>>>> set
>>>>
>>>>> or
>>>>>
>>>>>> the
>>>>>>>
>>>>>>>> default memory policy is overridden the output will miss
>>>>>>>>>>>
>>>>>>>>>> some of
>>>>
>>>>> the
>>>>>>
>>>>>>> rows.
>>>>>>>>>
>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>>>>>>>>>>>
>>>>>>>>>> parameter
>>>>>
>>>>>> was
>>>>>>>
>>>>>>>> set
>>>>>>>>>
>>>>>>>>>> explicitly by user code. Otherwise, the platform should be
>>>>>>>>>>>
>>>>>>>>>> wise
>>>>
>>>>> enough
>>>>>>>
>>>>>>>> to
>>>>>>>>>
>>>>>>>>>> instantiate it properly depending on the host memory usage.
>>>>>>>>>>>
>>>>>>>>>>> —
>>>>>>>>>>> Denis
>>>>>>>>>>>
>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>>>>>>>>>>>>
>>>>>>>>>>> gvvinblade@gmail.com>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The message without logging layout:
>>>>>>>>>>>>
>>>>>>>>>>>> Not enough memory for current process [required=29251MB,
>>>>>>>>>>>>
>>>>>>>>>>> available=15942MB].
>>>>>>>>>>>
>>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>>>>>>>>>>>>
>>>>>>>>>>> memory
>>>>
>>>>> allocated
>>>>>>>>>
>>>>>>>>>> for each node.
>>>>>>>>>>>>
>>>>>>>>>>>> Current settings:
>>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>
>>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>>>>>>>>>>>>
>>>>>>>>>>>> I think it's make sense to describe what these numbers are
>>>>>>>>>>>>
>>>>>>>>>>> consist
>>>>>>
>>>>>>> of.
>>>>>>>>
>>>>>>>>> We simple say which parameters have an impact on how much
>>>>>>>>>>>>
>>>>>>>>>>> memory
>>>>>
>>>>>> the
>>>>>>>
>>>>>>>> instance needs and their (parameters) actual values.
>>>>>>>>>>>>
>>>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
>>>>>>>>>>>>
>>>>>>>>>>> the
>>>>>
>>>>>> server
>>>>>>>>
>>>>>>>>> or
>>>>>>>>>>>
>>>>>>>>>>>> workstation and it also consumes memory.
>>>>>>>>>>>>
>>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>>>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>>>>>
>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
>>>>>>>>>>>>>
>>>>>>>>>>>> text?
>>>>>>
>>>>>>> Are
>>>>>>>>
>>>>>>>>> we
>>>>>>>>>
>>>>>>>>>> really going to print this whole thing out?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>>>>>>>>>>>>>
>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> This message appears on topology change in case the
>>>>>>>>>>>>>>
>>>>>>>>>>>>> available
>>>>
>>>>> memory
>>>>>>>>
>>>>>>>>> is
>>>>>>>>>
>>>>>>>>>> exceeded
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>>>>>>>>>>>>>>
>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>
>>>>>>>> :
>>>>>>>>>
>>>>>>>>>> An example of current impl:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> memory
>>>>>>
>>>>>>> for
>>>>>>>>
>>>>>>>>> current process [required=29251MB, available=15942MB].
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> change
>>>>
>>>>> MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> MemoryConfiguration.
>>>>>>
>>>>>>> defaultMemoryPolicySize
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to decrease memory allocated for each node.
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> settings:
>>>>>>
>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HeapInit=250MB
>>>>>>
>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HeapMax=3543MB
>>>>>>
>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DefaultMemoryPolicySize=
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 12753MB
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ignite
>>>>
>>>>> instanses
>>>>>>>>>
>>>>>>>>>> on
>>>>>>>>>>>>>
>>>>>>>>>>>>>> the server require: 12853MB
>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> internal.
>>>>>>>
>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> skozlov@gridgain.com
>>>>>>
>>>>>>> :
>>>>>>>>
>>>>>>>>> I suppose we should not forget JVM heap size and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> suggest to
>>>>
>>>>> reduce
>>>>>>>>
>>>>>>>>> both
>>>>>>>>>>>>>
>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> available
>>>>>>>
>>>>>>>> for
>>>>>>>>>
>>>>>>>>>> OS.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> memory
>>>>>
>>>>>> allocated
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> each node.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for
>>>>
>>>>> each
>>>>>>
>>>>>>> node
>>>>>>>>>
>>>>>>>>>> should
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> be? In that case we should suggest it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> GridHomePat
>>>>>>
>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> MB.
>>>>
>>>>> Use
>>>>>>
>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> change
>>>>>>
>>>>>>> the
>>>>>>>>
>>>>>>>>> setting.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> GridHomePat
>>>>>>
>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> memory
>>>>
>>>>> available
>>>>>>>>>>>>>
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> usage.
>>>>>
>>>>>> Use MemoryConfiguration and
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration
>>>>
>>>>> to
>>>>>
>>>>>> change
>>>>>>>>>>>>>
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> settings.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> available=15942MB]
>>>>
>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> GridHomePat
>>>>>>
>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> production
>>>>
>>>>> for
>>>>>
>>>>>> performance
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> deployment consistency reasons)
>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> GridHomePat
>>>>>>
>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ['ignite-sys-cache']]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> GridHomePat
>>>>>>
>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> te/incubator-ignite/libs/
>>>>
>>>>> licenses
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> yzhdanov@apache.org
>>>>>>>
>>>>>>>> :
>>>>>>>>>
>>>>>>>>>> Agree, let's release new version including tickets
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> mentioned
>>>>>>
>>>>>>> by
>>>>>>>>
>>>>>>>>> Denis
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ivan.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --Yakov
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Sergey Kozlov
>>>>>>>>>>>>>>>> GridGain Systems
>>>>>>>>>>>>>>>> www.gridgain.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by 李玉...@163,
18...@163.com.
Since it's an urgent bugfix version, why can't the version number be
defined as 2.1.1 or 2.1.4?
After all, functionality has not increased and documents do not require
big changes.
If the version number is 2.1 series, there is no need to publish a new
version of the document.
在 2017/8/28 下午10:19, Anton Vinogradov 写道:
> Igniters,
>
> Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> In this case all other issues with fixVersion = 2.2 should be moved to 2.3.
>
> Currently, I see 835 issues with fixVersion = 2.2
>
> Seems we should have only 4 issues with fixVersion = 2.2:
> - Change default max memory size from 80% to 20% -
> https://issues.apache.org/jira/browse/IGNITE-6182
> - Detecting low memory on ignite node startup -
> https://issues.apache.org/jira/browse/IGNITE-6003
> - Backport of improvements to checkpoint algorithm -
> https://issues.apache.org/jira/browse/IGNITE-????
> - ML profile is missing in 2.1 binary release -
> https://issues.apache.org/jira/browse/IGNITE-6193
>
> Please correct me in case I've missed something.
>
> Ivan,
>> Let's include to the release tickets with optimizations of checkpointing
> algorithm:
>> https://issues.apache.org/jira/browse/IGNITE-6178
>> https://issues.apache.org/jira/browse/IGNITE-6033
>> https://issues.apache.org/jira/browse/IGNITE-5961
>> This will help users who are experiencing problems with slow checkpoints.
>> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to
> soften "Ignite node crashed in the middle of checkpoint" message as per
> discussion <
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>> .
> All of these issues should have fixVersion 2.3.
> Please create backport issue and link it to all necessary issues.
>
> P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
>
> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <av...@gridgain.com>
> wrote:
>
>> Denis,
>>
>>> BTW, who is considered to be the release manager of this release?
>> I'll do it.
>>
>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gv...@gmail.com>
>> wrote:
>>
>>> Ok, the check happens at the node start time or on NODE_JOIN event
>>>
>>> in general it looks like:
>>>
>>> 1) calculate expected used memory = heap max + system cache max + all
>>> custom policies max + default policy size and put it into a node attribute
>>>
>>> 2) get total physycal memory, calculate expected safe to be used memory
>>> amount (leave 4 gb min or 20% of available memory for OS)
>>>
>>> 3) if expected used memory + expected used memory of other nodes on the
>>> host > than safe to be used memory amount, start calculating suggestions
>>>
>>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
>>> 100mb default polycy, if available memory is less we cannot suggest
>>> anything reasonable, print warning, stop calculation.
>>>
>>> 5) check heap size (shouldn't exceed 30% of available memory (total_memory
>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
>>> calculated value or 512MB minimal)
>>>
>>> 6) check if system cache size changed, suggest default value if it's so
>>>
>>> 7) in case 100 mb * policies count < available memory, suggest using
>>> default policy with max size equals to remaining memory (available - heap
>>> -
>>> system cache)
>>>
>>> 8) calculate new size for each memory policy ( it's user defined size *
>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>> remaining memory, devided by nodes number on the host or 100 mb minimal)
>>>
>>> 9) print suggestions
>>>
>>>
>>>
>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>
>>>> Igor, can you please describe the algorithm with all the thresholds?
>>>>
>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gvvinblade@gmail.com
>>>>
>>>> wrote:
>>>>
>>>>> The suggestion here is based on initial settings, and it's so because
>>>> there
>>>>> is no other nodes on the host in the example.
>>>>>
>>>>> The algorithm tries to preserve the original ratio of memory policies
>>>>> keeping numbers reasonable (for example after some thresshold it will
>>>>> suggest not to use several memory policies if there is not enough of
>>>> memory
>>>>> for all of them) and taking into consideration nodes count on the
>>> host,
>>>>> each jvm heap, needed memory for OS, etc
>>>>>
>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>>>
>>>>>> Looks good, but why in the example provided are we suggesting 8GB? 2
>>>>> nodes
>>>>>> with 8GB will completely exhaust the available memory. I would
>>> suggest
>>>> 6
>>>>> or
>>>>>> 7GB.
>>>>>>
>>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
>>> small.
>>>>>> Can you please comment?
>>>>>>
>>>>>> D.
>>>>>>
>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>>>> gvvinblade@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> One more example of possible warning:
>>>>>>>
>>>>>>> -----------------------------------------------------
>>>>>>> Excessive memory usage by Ignite node process (performance may
>>> drop)
>>>>>>> [requested=44613MB, available=15942MB].
>>>>>>>
>>>>>>> Please tune the folowing settings as suggested:
>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>>>>>>>
>>>>>>> Current settings:
>>>>>>> Java Heap maxSize: 3543MB
>>>>>>> Java Heap initSize: 250MB
>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>>>>>>> The overall expected memory usage by all Ignite nodes on the
>>> host:
>>>>>>> 44613MB
>>>>>>> -----------------------------------------------------
>>>>>>>
>>>>>>> Your thoughts?
>>>>>>>
>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>>>>>>>
>>>>>>>> Guys,
>>>>>>>>
>>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
>>>>> rolled
>>>>>>>> out in this emergency release:
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>>>>>>>>
>>>>>>>> Oleg, Yuri, please step in and handle the issue.
>>>>>>>>
>>>>>>>> BTW, who is considered to be the release manager of this
>>> release?
>>>>>>>> —
>>>>>>>> Denis
>>>>>>>>
>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org>
>>>>>>>> wrote:
>>>>>>>>> I like the format proposed by Denis, very clear.
>>>>>>>>>
>>>>>>>>> However, I also do not understand why a user should change the
>>>> size
>>>>>> of
>>>>>>>> some
>>>>>>>>> system cache. How would a user ever know what value to put
>>> there?
>>>>>> This
>>>>>>>>> value should be configured by Ignite automatically.
>>>>>>>>>
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>>> dmagda@apache.org>
>>>>>>> wrote:
>>>>>>>>>> Igor,
>>>>>>>>>>
>>>>>>>>>> Let me suggest this format.
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------
>>>>>>>>>> Excessive memory usage by Ignite node process (performance
>>> may
>>>>> drop)
>>>>>>>>>> [requested=29251MB, available=15942MB]
>>>>>>>>>>
>>>>>>>>>> Please tune the following settings:
>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
>>>> value]
>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
>>>>> suggested
>>>>>>>>>> value]
>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
>>>>> suggested
>>>>>>>>>> value]
>>>>>>>>>>
>>>>>>>>>> Current settings:
>>>>>>>>>> [DefaultMemoryPolicySize = value]
>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>> SystemCacheInitialSize = value
>>>>>>>>>> SystemCacheMaxSize = value
>>>>>>>>>> Java Heap Init Size = value
>>>>>>>>>> Java Heap Max Size = value
>>>>>>>>>>
>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>>> value
>>>>>>>>>> -------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Records in […] are optional. If custom memory policy is not
>>> set
>>>> or
>>>>>> the
>>>>>>>>>> default memory policy is overridden the output will miss
>>> some of
>>>>> the
>>>>>>>> rows.
>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>>>> parameter
>>>>>> was
>>>>>>>> set
>>>>>>>>>> explicitly by user code. Otherwise, the platform should be
>>> wise
>>>>>> enough
>>>>>>>> to
>>>>>>>>>> instantiate it properly depending on the host memory usage.
>>>>>>>>>>
>>>>>>>>>> —
>>>>>>>>>> Denis
>>>>>>>>>>
>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>>>>>> gvvinblade@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> The message without logging layout:
>>>>>>>>>>>
>>>>>>>>>>> Not enough memory for current process [required=29251MB,
>>>>>>>>>> available=15942MB].
>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>>> memory
>>>>>>>> allocated
>>>>>>>>>>> for each node.
>>>>>>>>>>>
>>>>>>>>>>> Current settings:
>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>
>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>>>>>>>>>>>
>>>>>>>>>>> I think it's make sense to describe what these numbers are
>>>>> consist
>>>>>>> of.
>>>>>>>>>>> We simple say which parameters have an impact on how much
>>>> memory
>>>>>> the
>>>>>>>>>>> instance needs and their (parameters) actual values.
>>>>>>>>>>>
>>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
>>>> the
>>>>>>> server
>>>>>>>>>> or
>>>>>>>>>>> workstation and it also consumes memory.
>>>>>>>>>>>
>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>>>>
>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
>>>>> text?
>>>>>>> Are
>>>>>>>> we
>>>>>>>>>>>> really going to print this whole thing out?
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> This message appears on topology change in case the
>>> available
>>>>>>> memory
>>>>>>>> is
>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>>>>>> gvvinblade@gmail.com
>>>>>>>> :
>>>>>>>>>>>>>> An example of current impl:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
>>>>> memory
>>>>>>> for
>>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>>> change
>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
>>>>> MemoryConfiguration.
>>>>>>>>>>>>> defaultMemoryPolicySize
>>>>>>>>>>>>>> to decrease memory allocated for each node.
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
>>>>> settings:
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>> HeapInit=250MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>> HeapMax=3543MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>> DefaultMemoryPolicySize=
>>>>>>>>>>>>>> 12753MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>>> ignite
>>>>>>>> instanses
>>>>>>>>>>>> on
>>>>>>>>>>>>>> the server require: 12853MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>>>>> skozlov@gridgain.com
>>>>>>> :
>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
>>> suggest to
>>>>>>> reduce
>>>>>>>>>>>> both
>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
>>>>>> available
>>>>>>>> for
>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
>>>> memory
>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> each node.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
>>> for
>>>>> each
>>>>>>>> node
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> be? In that case we should suggest it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
>>> MB.
>>>>> Use
>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
>>>>> change
>>>>>>> the
>>>>>>>>>>>>>>>> setting.
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>> memory
>>>>>>>>>>>> available
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
>>>> usage.
>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
>>> MemoryPolicyConfiguration
>>>> to
>>>>>>>>>>>> change
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> settings.
>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>>> available=15942MB]
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>>> production
>>>> for
>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> deployment consistency reasons)
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>>>>>>>>>> ['ignite-sys-cache']]
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>>> te/incubator-ignite/libs/
>>>>>>> licenses
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>>>>>> yzhdanov@apache.org
>>>>>>>> :
>>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
>>>>> mentioned
>>>>>>> by
>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> Ivan.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --Yakov
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Sergey Kozlov
>>>>>>>>>>>>>>> GridGain Systems
>>>>>>>>>>>>>>> www.gridgain.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by gvvinblade <gv...@gmail.com>.
To be honnest we check heap memory which is used by all Ignite nodes on the
host:
org.apache.ignite.internal.IgniteKernal#checkPhysicalRam
Shouldn't we do the same for offheap?
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
IGNITE-6128 is not merged yet. I already raised my concerns, please chime
in.
ср, 30 авг. 2017 г. в 21:45, Denis Magda <dm...@apache.org>:
> Have the all pre-release tests passed? If so, let’s kick off the vote.
>
> —
> Denis
>
> > On Aug 30, 2017, at 4:29 AM, Anton Vinogradov <av...@gridgain.com>
> wrote:
> >
> > 1) Fixes for:
> > - https://issues.apache.org/jira/browse/IGNITE-6204
> > - https://issues.apache.org/jira/browse/IGNITE-6193
> > merged
> >
> > 2) https://issues.apache.org/jira/browse/IGNITE-6003 resolved as
> duplicate
> > to https://issues.apache.org/jira/browse/IGNITE-6182
> >
> > 3) Current list if issues related to release 2.2:
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Ignite%20and%20fixVersion%20%3D%202.2
> >
> > On Wed, Aug 30, 2017 at 1:24 PM, Vladimir Ozerov <vo...@gridgain.com>
> > wrote:
> >
> >> I looked at the code of IGNITE-6182 (memory policy suggestions), and I
> do
> >> not like it. Too complex. My proposal:
> >> 1) Do not take in count JVM heap. First, we do not need whether it will
> be
> >> really used or not. Second, we never checked it in heap-based Ignite
> 1.x,
> >> and never had a single compliant.
> >> 2) We need to sum up all memory policies and *checkpoint buffer size*.
> If
> >> it exceeds 80% of machine's RAM - print a warning
> >>
> >> On Mon, Aug 28, 2017 at 7:26 PM, Denis Magda <dm...@apache.org> wrote:
> >>
> >>> Anton, thanks for stepping in! The tickets set looks complete for me!
> >>>
> >>> Let’s start the vote once the fixes are merged and tests pass.
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On Aug 28, 2017, at 7:19 AM, Anton Vinogradov <
> >> avinogradov@gridgain.com>
> >>> wrote:
> >>>>
> >>>> Igniters,
> >>>>
> >>>> Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> >>>> In this case all other issues with fixVersion = 2.2 should be moved to
> >>> 2.3.
> >>>>
> >>>> Currently, I see 835 issues with fixVersion = 2.2
> >>>>
> >>>> Seems we should have only 4 issues with fixVersion = 2.2:
> >>>> - Change default max memory size from 80% to 20% -
> >>>> https://issues.apache.org/jira/browse/IGNITE-6182
> >>>> - Detecting low memory on ignite node startup -
> >>>> https://issues.apache.org/jira/browse/IGNITE-6003
> >>>> - Backport of improvements to checkpoint algorithm -
> >>>> https://issues.apache.org/jira/browse/IGNITE-????
> >>>> - ML profile is missing in 2.1 binary release -
> >>>> https://issues.apache.org/jira/browse/IGNITE-6193
> >>>>
> >>>> Please correct me in case I've missed something.
> >>>>
> >>>> Ivan,
> >>>>> Let's include to the release tickets with optimizations of
> >> checkpointing
> >>>> algorithm:
> >>>>> https://issues.apache.org/jira/browse/IGNITE-6178
> >>>>> https://issues.apache.org/jira/browse/IGNITE-6033
> >>>>> https://issues.apache.org/jira/browse/IGNITE-5961
> >>>>> This will help users who are experiencing problems with slow
> >>> checkpoints.
> >>>>> Also, let's include
> https://issues.apache.org/jira/browse/IGNITE-6183
> >>> to
> >>>> soften "Ignite node crashed in the middle of checkpoint" message as
> per
> >>>> discussion <
> >>>> http://apache-ignite-developers.2346864.n4.nabble.
> >>> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
> >> false-td20473.html
> >>>>> .
> >>>>
> >>>> All of these issues should have fixVersion 2.3.
> >>>> Please create backport issue and link it to all necessary issues.
> >>>>
> >>>> P.s. I've created branch ignite-2.2, currently it equals to
> ignite-2.1.
> >>>>
> >>>> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
> >>> avinogradov@gridgain.com>
> >>>> wrote:
> >>>>
> >>>>> Denis,
> >>>>>
> >>>>>> BTW, who is considered to be the release manager of this release?
> >>>>> I'll do it.
> >>>>>
> >>>>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
> >> gvvinblade@gmail.com
> >>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Ok, the check happens at the node start time or on NODE_JOIN event
> >>>>>>
> >>>>>> in general it looks like:
> >>>>>>
> >>>>>> 1) calculate expected used memory = heap max + system cache max +
> all
> >>>>>> custom policies max + default policy size and put it into a node
> >>> attribute
> >>>>>>
> >>>>>> 2) get total physycal memory, calculate expected safe to be used
> >> memory
> >>>>>> amount (leave 4 gb min or 20% of available memory for OS)
> >>>>>>
> >>>>>> 3) if expected used memory + expected used memory of other nodes on
> >> the
> >>>>>> host > than safe to be used memory amount, start calculating
> >>> suggestions
> >>>>>>
> >>>>>> 4) Each ignite instance needs at least 512mb heap + 40mb system
> >> cache +
> >>>>>> 100mb default polycy, if available memory is less we cannot suggest
> >>>>>> anything reasonable, print warning, stop calculation.
> >>>>>>
> >>>>>> 5) check heap size (shouldn't exceed 30% of available memory
> >>> (total_memory
> >>>>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
> >>> just
> >>>>>> calculated value or 512MB minimal)
> >>>>>>
> >>>>>> 6) check if system cache size changed, suggest default value if it's
> >> so
> >>>>>>
> >>>>>> 7) in case 100 mb * policies count < available memory, suggest using
> >>>>>> default policy with max size equals to remaining memory (available -
> >>> heap
> >>>>>> -
> >>>>>> system cache)
> >>>>>>
> >>>>>> 8) calculate new size for each memory policy ( it's user defined
> >> size *
> >>>>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> >>>>>> remaining memory, devided by nodes number on the host or 100 mb
> >>> minimal)
> >>>>>>
> >>>>>> 9) print suggestions
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
> >>> :
> >>>>>>
> >>>>>>> Igor, can you please describe the algorithm with all the
> thresholds?
> >>>>>>>
> >>>>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
> >>> gvvinblade@gmail.com
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> The suggestion here is based on initial settings, and it's so
> >> because
> >>>>>>> there
> >>>>>>>> is no other nodes on the host in the example.
> >>>>>>>>
> >>>>>>>> The algorithm tries to preserve the original ratio of memory
> >> policies
> >>>>>>>> keeping numbers reasonable (for example after some thresshold it
> >> will
> >>>>>>>> suggest not to use several memory policies if there is not enough
> >> of
> >>>>>>> memory
> >>>>>>>> for all of them) and taking into consideration nodes count on the
> >>>>>> host,
> >>>>>>>> each jvm heap, needed memory for OS, etc
> >>>>>>>>
> >>>>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
> >> dsetrakyan@apache.org
> >>>> :
> >>>>>>>>
> >>>>>>>>> Looks good, but why in the example provided are we suggesting
> >> 8GB? 2
> >>>>>>>> nodes
> >>>>>>>>> with 8GB will completely exhaust the available memory. I would
> >>>>>> suggest
> >>>>>>> 6
> >>>>>>>> or
> >>>>>>>>> 7GB.
> >>>>>>>>>
> >>>>>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
> >>>>>> small.
> >>>>>>>>>
> >>>>>>>>> Can you please comment?
> >>>>>>>>>
> >>>>>>>>> D.
> >>>>>>>>>
> >>>>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> >>>>>>> gvvinblade@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> One more example of possible warning:
> >>>>>>>>>>
> >>>>>>>>>> -----------------------------------------------------
> >>>>>>>>>> Excessive memory usage by Ignite node process (performance may
> >>>>>> drop)
> >>>>>>>>>> [requested=44613MB, available=15942MB].
> >>>>>>>>>>
> >>>>>>>>>> Please tune the folowing settings as suggested:
> >>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> >>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> >>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> >>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> >>>>>>>>>>
> >>>>>>>>>> Current settings:
> >>>>>>>>>> Java Heap maxSize: 3543MB
> >>>>>>>>>> Java Heap initSize: 250MB
> >>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> >>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> >>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> >>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> >>>>>>>>>> The overall expected memory usage by all Ignite nodes on the
> >>>>>> host:
> >>>>>>>>>> 44613MB
> >>>>>>>>>> -----------------------------------------------------
> >>>>>>>>>>
> >>>>>>>>>> Your thoughts?
> >>>>>>>>>>
> >>>>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> >>>>>>>>>>
> >>>>>>>>>>> Guys,
> >>>>>>>>>>>
> >>>>>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed
> and
> >>>>>>>> rolled
> >>>>>>>>>>> out in this emergency release:
> >>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
> >>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
> >>>>>>>>>>>
> >>>>>>>>>>> Oleg, Yuri, please step in and handle the issue.
> >>>>>>>>>>>
> >>>>>>>>>>> BTW, who is considered to be the release manager of this
> >>>>>> release?
> >>>>>>>>>>>
> >>>>>>>>>>> —
> >>>>>>>>>>> Denis
> >>>>>>>>>>>
> >>>>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> >>>>>>>>> dsetrakyan@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I like the format proposed by Denis, very clear.
> >>>>>>>>>>>>
> >>>>>>>>>>>> However, I also do not understand why a user should change the
> >>>>>>> size
> >>>>>>>>> of
> >>>>>>>>>>> some
> >>>>>>>>>>>> system cache. How would a user ever know what value to put
> >>>>>> there?
> >>>>>>>>> This
> >>>>>>>>>>>> value should be configured by Ignite automatically.
> >>>>>>>>>>>>
> >>>>>>>>>>>> D.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> >>>>>> dmagda@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Igor,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Let me suggest this format.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ---------------------------------------------
> >>>>>>>>>>>>> Excessive memory usage by Ignite node process (performance
> >>>>>> may
> >>>>>>>> drop)
> >>>>>>>>>>>>> [requested=29251MB, available=15942MB]
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Please tune the following settings:
> >>>>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
> >>>>>>> value]
> >>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
> >>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> >>>>>>>> suggested
> >>>>>>>>>>>>> value]
> >>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> >>>>>>>> suggested
> >>>>>>>>>>>>> value]
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Current settings:
> >>>>>>>>>>>>> [DefaultMemoryPolicySize = value]
> >>>>>>>>>>>>> [{policy_name_1} size = value]
> >>>>>>>>>>>>> [{policy_name_1} size = value]
> >>>>>>>>>>>>> SystemCacheInitialSize = value
> >>>>>>>>>>>>> SystemCacheMaxSize = value
> >>>>>>>>>>>>> Java Heap Init Size = value
> >>>>>>>>>>>>> Java Heap Max Size = value
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
> >>>>>> value
> >>>>>>>>>>>>> -------------------------------------------
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Records in […] are optional. If custom memory policy is not
> >>>>>> set
> >>>>>>> or
> >>>>>>>>> the
> >>>>>>>>>>>>> default memory policy is overridden the output will miss
> >>>>>> some of
> >>>>>>>> the
> >>>>>>>>>>> rows.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
> >>>>>>> parameter
> >>>>>>>>> was
> >>>>>>>>>>> set
> >>>>>>>>>>>>> explicitly by user code. Otherwise, the platform should be
> >>>>>> wise
> >>>>>>>>> enough
> >>>>>>>>>>> to
> >>>>>>>>>>>>> instantiate it properly depending on the host memory usage.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> —
> >>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> >>>>>>>>> gvvinblade@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The message without logging layout:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Not enough memory for current process [required=29251MB,
> >>>>>>>>>>>>> available=15942MB].
> >>>>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
> >>>>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> >>>>>> memory
> >>>>>>>>>>> allocated
> >>>>>>>>>>>>>> for each node.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Current settings:
> >>>>>>>>>>>>>> HeapInit=250MB
> >>>>>>>>>>>>>> HeapMax=3543MB
> >>>>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
> >>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> >>>>>>>>>>>>>> SystemCacheMaxSize=100MB
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think it's make sense to describe what these numbers are
> >>>>>>>> consist
> >>>>>>>>>> of.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We simple say which parameters have an impact on how much
> >>>>>>> memory
> >>>>>>>>> the
> >>>>>>>>>>>>>> instance needs and their (parameters) actual values.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
> >>>>>>> the
> >>>>>>>>>> server
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>>> workstation and it also consumes memory.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> >>>>>>>>>>>>>> dsetrakyan@apache.org> написал:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
> >>>>>>>> text?
> >>>>>>>>>> Are
> >>>>>>>>>>> we
> >>>>>>>>>>>>>>> really going to print this whole thing out?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> >>>>>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This message appears on topology change in case the
> >>>>>> available
> >>>>>>>>>> memory
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>> exceeded
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> >>>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>> :
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> An example of current impl:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
> >>>>>>>> memory
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> >>>>>> change
> >>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
> >>>>>>>> MemoryConfiguration.
> >>>>>>>>>>>>>>>> defaultMemoryPolicySize
> >>>>>>>>>>>>>>>>> to decrease memory allocated for each node.
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> >>>>>>>> settings:
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>> HeapInit=250MB
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>> HeapMax=3543MB
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>> DefaultMemoryPolicySize=
> >>>>>>>>>>>>>>>>> 12753MB
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> >>>>>> ignite
> >>>>>>>>>>> instanses
> >>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>> the server require: 12853MB
> >>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>>>>> internal.
> >>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> >>>>>>>> skozlov@gridgain.com
> >>>>>>>>>> :
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
> >>>>>> suggest to
> >>>>>>>>>> reduce
> >>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>> options
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>>>>>> dsetrakyan@apache.org
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
> >>>>>>>>> available
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>> OS.
> >>>>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> >>>>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
> >>>>>>> memory
> >>>>>>>>>>>>>>>> allocated
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> each node.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
> >>>>>> for
> >>>>>>>> each
> >>>>>>>>>>> node
> >>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>> be? In that case we should suggest it.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> >>>>>>>>>>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
> >>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
> >>>>>> MB.
> >>>>>>>> Use
> >>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
> >>>>>>>> change
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> setting.
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
> >>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> >>>>>> memory
> >>>>>>>>>>>>>>> available
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> OS.
> >>>>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
> >>>>>>> usage.
> >>>>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
> >>>>>> MemoryPolicyConfiguration
> >>>>>>> to
> >>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> settings.
> >>>>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
> >>>>>> available=15942MB]
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
> >>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
> >>>>>> production
> >>>>>>> for
> >>>>>>>>>>>>>>>>>> performance
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> deployment consistency reasons)
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
> >>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> >>>>>>>>>>>>>>>> ['ignite-sys-cache']]
> >>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
> >>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>>>>> GridHomePat
> >>>>>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
> >>>>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
> >>>>>> te/incubator-ignite/libs/
> >>>>>>>>>> licenses
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> >>>>>>>>> yzhdanov@apache.org
> >>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
> >>>>>>>> mentioned
> >>>>>>>>>> by
> >>>>>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> Ivan.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> --Yakov
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Sergey Kozlov
> >>>>>>>>>>>>>>>>>> GridGain Systems
> >>>>>>>>>>>>>>>>>> www.gridgain.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Denis Magda <dm...@apache.org>.
Have the all pre-release tests passed? If so, let’s kick off the vote.
—
Denis
> On Aug 30, 2017, at 4:29 AM, Anton Vinogradov <av...@gridgain.com> wrote:
>
> 1) Fixes for:
> - https://issues.apache.org/jira/browse/IGNITE-6204
> - https://issues.apache.org/jira/browse/IGNITE-6193
> merged
>
> 2) https://issues.apache.org/jira/browse/IGNITE-6003 resolved as duplicate
> to https://issues.apache.org/jira/browse/IGNITE-6182
>
> 3) Current list if issues related to release 2.2:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Ignite%20and%20fixVersion%20%3D%202.2
>
> On Wed, Aug 30, 2017 at 1:24 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
>> I looked at the code of IGNITE-6182 (memory policy suggestions), and I do
>> not like it. Too complex. My proposal:
>> 1) Do not take in count JVM heap. First, we do not need whether it will be
>> really used or not. Second, we never checked it in heap-based Ignite 1.x,
>> and never had a single compliant.
>> 2) We need to sum up all memory policies and *checkpoint buffer size*. If
>> it exceeds 80% of machine's RAM - print a warning
>>
>> On Mon, Aug 28, 2017 at 7:26 PM, Denis Magda <dm...@apache.org> wrote:
>>
>>> Anton, thanks for stepping in! The tickets set looks complete for me!
>>>
>>> Let’s start the vote once the fixes are merged and tests pass.
>>>
>>> —
>>> Denis
>>>
>>>> On Aug 28, 2017, at 7:19 AM, Anton Vinogradov <
>> avinogradov@gridgain.com>
>>> wrote:
>>>>
>>>> Igniters,
>>>>
>>>> Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
>>>> In this case all other issues with fixVersion = 2.2 should be moved to
>>> 2.3.
>>>>
>>>> Currently, I see 835 issues with fixVersion = 2.2
>>>>
>>>> Seems we should have only 4 issues with fixVersion = 2.2:
>>>> - Change default max memory size from 80% to 20% -
>>>> https://issues.apache.org/jira/browse/IGNITE-6182
>>>> - Detecting low memory on ignite node startup -
>>>> https://issues.apache.org/jira/browse/IGNITE-6003
>>>> - Backport of improvements to checkpoint algorithm -
>>>> https://issues.apache.org/jira/browse/IGNITE-????
>>>> - ML profile is missing in 2.1 binary release -
>>>> https://issues.apache.org/jira/browse/IGNITE-6193
>>>>
>>>> Please correct me in case I've missed something.
>>>>
>>>> Ivan,
>>>>> Let's include to the release tickets with optimizations of
>> checkpointing
>>>> algorithm:
>>>>> https://issues.apache.org/jira/browse/IGNITE-6178
>>>>> https://issues.apache.org/jira/browse/IGNITE-6033
>>>>> https://issues.apache.org/jira/browse/IGNITE-5961
>>>>> This will help users who are experiencing problems with slow
>>> checkpoints.
>>>>> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183
>>> to
>>>> soften "Ignite node crashed in the middle of checkpoint" message as per
>>>> discussion <
>>>> http://apache-ignite-developers.2346864.n4.nabble.
>>> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
>> false-td20473.html
>>>>> .
>>>>
>>>> All of these issues should have fixVersion 2.3.
>>>> Please create backport issue and link it to all necessary issues.
>>>>
>>>> P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
>>>>
>>>> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
>>> avinogradov@gridgain.com>
>>>> wrote:
>>>>
>>>>> Denis,
>>>>>
>>>>>> BTW, who is considered to be the release manager of this release?
>>>>> I'll do it.
>>>>>
>>>>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
>> gvvinblade@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>>> Ok, the check happens at the node start time or on NODE_JOIN event
>>>>>>
>>>>>> in general it looks like:
>>>>>>
>>>>>> 1) calculate expected used memory = heap max + system cache max + all
>>>>>> custom policies max + default policy size and put it into a node
>>> attribute
>>>>>>
>>>>>> 2) get total physycal memory, calculate expected safe to be used
>> memory
>>>>>> amount (leave 4 gb min or 20% of available memory for OS)
>>>>>>
>>>>>> 3) if expected used memory + expected used memory of other nodes on
>> the
>>>>>> host > than safe to be used memory amount, start calculating
>>> suggestions
>>>>>>
>>>>>> 4) Each ignite instance needs at least 512mb heap + 40mb system
>> cache +
>>>>>> 100mb default polycy, if available memory is less we cannot suggest
>>>>>> anything reasonable, print warning, stop calculation.
>>>>>>
>>>>>> 5) check heap size (shouldn't exceed 30% of available memory
>>> (total_memory
>>>>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
>>> just
>>>>>> calculated value or 512MB minimal)
>>>>>>
>>>>>> 6) check if system cache size changed, suggest default value if it's
>> so
>>>>>>
>>>>>> 7) in case 100 mb * policies count < available memory, suggest using
>>>>>> default policy with max size equals to remaining memory (available -
>>> heap
>>>>>> -
>>>>>> system cache)
>>>>>>
>>>>>> 8) calculate new size for each memory policy ( it's user defined
>> size *
>>>>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>>>>> remaining memory, devided by nodes number on the host or 100 mb
>>> minimal)
>>>>>>
>>>>>> 9) print suggestions
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
>>> :
>>>>>>
>>>>>>> Igor, can you please describe the algorithm with all the thresholds?
>>>>>>>
>>>>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
>>> gvvinblade@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The suggestion here is based on initial settings, and it's so
>> because
>>>>>>> there
>>>>>>>> is no other nodes on the host in the example.
>>>>>>>>
>>>>>>>> The algorithm tries to preserve the original ratio of memory
>> policies
>>>>>>>> keeping numbers reasonable (for example after some thresshold it
>> will
>>>>>>>> suggest not to use several memory policies if there is not enough
>> of
>>>>>>> memory
>>>>>>>> for all of them) and taking into consideration nodes count on the
>>>>>> host,
>>>>>>>> each jvm heap, needed memory for OS, etc
>>>>>>>>
>>>>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
>> dsetrakyan@apache.org
>>>> :
>>>>>>>>
>>>>>>>>> Looks good, but why in the example provided are we suggesting
>> 8GB? 2
>>>>>>>> nodes
>>>>>>>>> with 8GB will completely exhaust the available memory. I would
>>>>>> suggest
>>>>>>> 6
>>>>>>>> or
>>>>>>>>> 7GB.
>>>>>>>>>
>>>>>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
>>>>>> small.
>>>>>>>>>
>>>>>>>>> Can you please comment?
>>>>>>>>>
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>>>>>>> gvvinblade@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> One more example of possible warning:
>>>>>>>>>>
>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>> Excessive memory usage by Ignite node process (performance may
>>>>>> drop)
>>>>>>>>>> [requested=44613MB, available=15942MB].
>>>>>>>>>>
>>>>>>>>>> Please tune the folowing settings as suggested:
>>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>>>>>>>>>>
>>>>>>>>>> Current settings:
>>>>>>>>>> Java Heap maxSize: 3543MB
>>>>>>>>>> Java Heap initSize: 250MB
>>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>>>>>>>>>> The overall expected memory usage by all Ignite nodes on the
>>>>>> host:
>>>>>>>>>> 44613MB
>>>>>>>>>> -----------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Your thoughts?
>>>>>>>>>>
>>>>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>>>>>>>>>>
>>>>>>>>>>> Guys,
>>>>>>>>>>>
>>>>>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
>>>>>>>> rolled
>>>>>>>>>>> out in this emergency release:
>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>>>>>>>>>>>
>>>>>>>>>>> Oleg, Yuri, please step in and handle the issue.
>>>>>>>>>>>
>>>>>>>>>>> BTW, who is considered to be the release manager of this
>>>>>> release?
>>>>>>>>>>>
>>>>>>>>>>> —
>>>>>>>>>>> Denis
>>>>>>>>>>>
>>>>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>>>>>>>>> dsetrakyan@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I like the format proposed by Denis, very clear.
>>>>>>>>>>>>
>>>>>>>>>>>> However, I also do not understand why a user should change the
>>>>>>> size
>>>>>>>>> of
>>>>>>>>>>> some
>>>>>>>>>>>> system cache. How would a user ever know what value to put
>>>>>> there?
>>>>>>>>> This
>>>>>>>>>>>> value should be configured by Ignite automatically.
>>>>>>>>>>>>
>>>>>>>>>>>> D.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>>>>>> dmagda@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Igor,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me suggest this format.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>> Excessive memory usage by Ignite node process (performance
>>>>>> may
>>>>>>>> drop)
>>>>>>>>>>>>> [requested=29251MB, available=15942MB]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please tune the following settings:
>>>>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
>>>>>>> value]
>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
>>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
>>>>>>>> suggested
>>>>>>>>>>>>> value]
>>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
>>>>>>>> suggested
>>>>>>>>>>>>> value]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>> [DefaultMemoryPolicySize = value]
>>>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>>>> SystemCacheInitialSize = value
>>>>>>>>>>>>> SystemCacheMaxSize = value
>>>>>>>>>>>>> Java Heap Init Size = value
>>>>>>>>>>>>> Java Heap Max Size = value
>>>>>>>>>>>>>
>>>>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>>>>>> value
>>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Records in […] are optional. If custom memory policy is not
>>>>>> set
>>>>>>> or
>>>>>>>>> the
>>>>>>>>>>>>> default memory policy is overridden the output will miss
>>>>>> some of
>>>>>>>> the
>>>>>>>>>>> rows.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>>>>>>> parameter
>>>>>>>>> was
>>>>>>>>>>> set
>>>>>>>>>>>>> explicitly by user code. Otherwise, the platform should be
>>>>>> wise
>>>>>>>>> enough
>>>>>>>>>>> to
>>>>>>>>>>>>> instantiate it properly depending on the host memory usage.
>>>>>>>>>>>>>
>>>>>>>>>>>>> —
>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>>>>>>>>> gvvinblade@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The message without logging layout:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Not enough memory for current process [required=29251MB,
>>>>>>>>>>>>> available=15942MB].
>>>>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>>>>>> memory
>>>>>>>>>>> allocated
>>>>>>>>>>>>>> for each node.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think it's make sense to describe what these numbers are
>>>>>>>> consist
>>>>>>>>>> of.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We simple say which parameters have an impact on how much
>>>>>>> memory
>>>>>>>>> the
>>>>>>>>>>>>>> instance needs and their (parameters) actual values.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
>>>>>>> the
>>>>>>>>>> server
>>>>>>>>>>>>> or
>>>>>>>>>>>>>> workstation and it also consumes memory.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>>>>>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
>>>>>>>> text?
>>>>>>>>>> Are
>>>>>>>>>>> we
>>>>>>>>>>>>>>> really going to print this whole thing out?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This message appears on topology change in case the
>>>>>> available
>>>>>>>>>> memory
>>>>>>>>>>> is
>>>>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> An example of current impl:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
>>>>>>>> memory
>>>>>>>>>> for
>>>>>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>>>>>> change
>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
>>>>>>>> MemoryConfiguration.
>>>>>>>>>>>>>>>> defaultMemoryPolicySize
>>>>>>>>>>>>>>>>> to decrease memory allocated for each node.
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
>>>>>>>> settings:
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>> HeapInit=250MB
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>> HeapMax=3543MB
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>> DefaultMemoryPolicySize=
>>>>>>>>>>>>>>>>> 12753MB
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>>>>>> ignite
>>>>>>>>>>> instanses
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> the server require: 12853MB
>>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>>> internal.
>>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>>>>>>>> skozlov@gridgain.com
>>>>>>>>>> :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
>>>>>> suggest to
>>>>>>>>>> reduce
>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
>>>>>>>>> available
>>>>>>>>>>> for
>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>>>>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
>>>>>>> memory
>>>>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> each node.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
>>>>>> for
>>>>>>>> each
>>>>>>>>>>> node
>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>> be? In that case we should suggest it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
>>>>>> MB.
>>>>>>>> Use
>>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
>>>>>>>> change
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> setting.
>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>>>>> memory
>>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
>>>>>>> usage.
>>>>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
>>>>>> MemoryPolicyConfiguration
>>>>>>> to
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> settings.
>>>>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>>>>>> available=15942MB]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>>>>>> production
>>>>>>> for
>>>>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> deployment consistency reasons)
>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>>>>>>>>>>>>> ['ignite-sys-cache']]
>>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>>>>>> te/incubator-ignite/libs/
>>>>>>>>>> licenses
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>>>>>>>>> yzhdanov@apache.org
>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
>>>>>>>> mentioned
>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> Ivan.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --Yakov
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Sergey Kozlov
>>>>>>>>>>>>>>>>>> GridGain Systems
>>>>>>>>>>>>>>>>>> www.gridgain.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Anton Vinogradov <av...@gridgain.com>.
1) Fixes for:
- https://issues.apache.org/jira/browse/IGNITE-6204
- https://issues.apache.org/jira/browse/IGNITE-6193
merged
2) https://issues.apache.org/jira/browse/IGNITE-6003 resolved as duplicate
to https://issues.apache.org/jira/browse/IGNITE-6182
3) Current list if issues related to release 2.2:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20Ignite%20and%20fixVersion%20%3D%202.2
On Wed, Aug 30, 2017 at 1:24 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> I looked at the code of IGNITE-6182 (memory policy suggestions), and I do
> not like it. Too complex. My proposal:
> 1) Do not take in count JVM heap. First, we do not need whether it will be
> really used or not. Second, we never checked it in heap-based Ignite 1.x,
> and never had a single compliant.
> 2) We need to sum up all memory policies and *checkpoint buffer size*. If
> it exceeds 80% of machine's RAM - print a warning
>
> On Mon, Aug 28, 2017 at 7:26 PM, Denis Magda <dm...@apache.org> wrote:
>
> > Anton, thanks for stepping in! The tickets set looks complete for me!
> >
> > Let’s start the vote once the fixes are merged and tests pass.
> >
> > —
> > Denis
> >
> > > On Aug 28, 2017, at 7:19 AM, Anton Vinogradov <
> avinogradov@gridgain.com>
> > wrote:
> > >
> > > Igniters,
> > >
> > > Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> > > In this case all other issues with fixVersion = 2.2 should be moved to
> > 2.3.
> > >
> > > Currently, I see 835 issues with fixVersion = 2.2
> > >
> > > Seems we should have only 4 issues with fixVersion = 2.2:
> > > - Change default max memory size from 80% to 20% -
> > > https://issues.apache.org/jira/browse/IGNITE-6182
> > > - Detecting low memory on ignite node startup -
> > > https://issues.apache.org/jira/browse/IGNITE-6003
> > > - Backport of improvements to checkpoint algorithm -
> > > https://issues.apache.org/jira/browse/IGNITE-????
> > > - ML profile is missing in 2.1 binary release -
> > > https://issues.apache.org/jira/browse/IGNITE-6193
> > >
> > > Please correct me in case I've missed something.
> > >
> > > Ivan,
> > >> Let's include to the release tickets with optimizations of
> checkpointing
> > > algorithm:
> > >> https://issues.apache.org/jira/browse/IGNITE-6178
> > >> https://issues.apache.org/jira/browse/IGNITE-6033
> > >> https://issues.apache.org/jira/browse/IGNITE-5961
> > >> This will help users who are experiencing problems with slow
> > checkpoints.
> > >> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183
> > to
> > > soften "Ignite node crashed in the middle of checkpoint" message as per
> > > discussion <
> > > http://apache-ignite-developers.2346864.n4.nabble.
> > com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-
> false-td20473.html
> > >> .
> > >
> > > All of these issues should have fixVersion 2.3.
> > > Please create backport issue and link it to all necessary issues.
> > >
> > > P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
> > >
> > > On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
> > avinogradov@gridgain.com>
> > > wrote:
> > >
> > >> Denis,
> > >>
> > >>> BTW, who is considered to be the release manager of this release?
> > >> I'll do it.
> > >>
> > >> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <
> gvvinblade@gmail.com
> > >
> > >> wrote:
> > >>
> > >>> Ok, the check happens at the node start time or on NODE_JOIN event
> > >>>
> > >>> in general it looks like:
> > >>>
> > >>> 1) calculate expected used memory = heap max + system cache max + all
> > >>> custom policies max + default policy size and put it into a node
> > attribute
> > >>>
> > >>> 2) get total physycal memory, calculate expected safe to be used
> memory
> > >>> amount (leave 4 gb min or 20% of available memory for OS)
> > >>>
> > >>> 3) if expected used memory + expected used memory of other nodes on
> the
> > >>> host > than safe to be used memory amount, start calculating
> > suggestions
> > >>>
> > >>> 4) Each ignite instance needs at least 512mb heap + 40mb system
> cache +
> > >>> 100mb default polycy, if available memory is less we cannot suggest
> > >>> anything reasonable, print warning, stop calculation.
> > >>>
> > >>> 5) check heap size (shouldn't exceed 30% of available memory
> > (total_memory
> > >>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
> > just
> > >>> calculated value or 512MB minimal)
> > >>>
> > >>> 6) check if system cache size changed, suggest default value if it's
> so
> > >>>
> > >>> 7) in case 100 mb * policies count < available memory, suggest using
> > >>> default policy with max size equals to remaining memory (available -
> > heap
> > >>> -
> > >>> system cache)
> > >>>
> > >>> 8) calculate new size for each memory policy ( it's user defined
> size *
> > >>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> > >>> remaining memory, devided by nodes number on the host or 100 mb
> > minimal)
> > >>>
> > >>> 9) print suggestions
> > >>>
> > >>>
> > >>>
> > >>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
> >:
> > >>>
> > >>>> Igor, can you please describe the algorithm with all the thresholds?
> > >>>>
> > >>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
> > gvvinblade@gmail.com
> > >>>>
> > >>>> wrote:
> > >>>>
> > >>>>> The suggestion here is based on initial settings, and it's so
> because
> > >>>> there
> > >>>>> is no other nodes on the host in the example.
> > >>>>>
> > >>>>> The algorithm tries to preserve the original ratio of memory
> policies
> > >>>>> keeping numbers reasonable (for example after some thresshold it
> will
> > >>>>> suggest not to use several memory policies if there is not enough
> of
> > >>>> memory
> > >>>>> for all of them) and taking into consideration nodes count on the
> > >>> host,
> > >>>>> each jvm heap, needed memory for OS, etc
> > >>>>>
> > >>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > >:
> > >>>>>
> > >>>>>> Looks good, but why in the example provided are we suggesting
> 8GB? 2
> > >>>>> nodes
> > >>>>>> with 8GB will completely exhaust the available memory. I would
> > >>> suggest
> > >>>> 6
> > >>>>> or
> > >>>>>> 7GB.
> > >>>>>>
> > >>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
> > >>> small.
> > >>>>>>
> > >>>>>> Can you please comment?
> > >>>>>>
> > >>>>>> D.
> > >>>>>>
> > >>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> > >>>> gvvinblade@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> One more example of possible warning:
> > >>>>>>>
> > >>>>>>> -----------------------------------------------------
> > >>>>>>> Excessive memory usage by Ignite node process (performance may
> > >>> drop)
> > >>>>>>> [requested=44613MB, available=15942MB].
> > >>>>>>>
> > >>>>>>> Please tune the folowing settings as suggested:
> > >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > >>>>>>>
> > >>>>>>> Current settings:
> > >>>>>>> Java Heap maxSize: 3543MB
> > >>>>>>> Java Heap initSize: 250MB
> > >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > >>>>>>> The overall expected memory usage by all Ignite nodes on the
> > >>> host:
> > >>>>>>> 44613MB
> > >>>>>>> -----------------------------------------------------
> > >>>>>>>
> > >>>>>>> Your thoughts?
> > >>>>>>>
> > >>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> > >>>>>>>
> > >>>>>>>> Guys,
> > >>>>>>>>
> > >>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
> > >>>>> rolled
> > >>>>>>>> out in this emergency release:
> > >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
> > >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
> > >>>>>>>>
> > >>>>>>>> Oleg, Yuri, please step in and handle the issue.
> > >>>>>>>>
> > >>>>>>>> BTW, who is considered to be the release manager of this
> > >>> release?
> > >>>>>>>>
> > >>>>>>>> —
> > >>>>>>>> Denis
> > >>>>>>>>
> > >>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > >>>>>> dsetrakyan@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> I like the format proposed by Denis, very clear.
> > >>>>>>>>>
> > >>>>>>>>> However, I also do not understand why a user should change the
> > >>>> size
> > >>>>>> of
> > >>>>>>>> some
> > >>>>>>>>> system cache. How would a user ever know what value to put
> > >>> there?
> > >>>>>> This
> > >>>>>>>>> value should be configured by Ignite automatically.
> > >>>>>>>>>
> > >>>>>>>>> D.
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> > >>> dmagda@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Igor,
> > >>>>>>>>>>
> > >>>>>>>>>> Let me suggest this format.
> > >>>>>>>>>>
> > >>>>>>>>>> ---------------------------------------------
> > >>>>>>>>>> Excessive memory usage by Ignite node process (performance
> > >>> may
> > >>>>> drop)
> > >>>>>>>>>> [requested=29251MB, available=15942MB]
> > >>>>>>>>>>
> > >>>>>>>>>> Please tune the following settings:
> > >>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
> > >>>> value]
> > >>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
> > >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> > >>>>> suggested
> > >>>>>>>>>> value]
> > >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> > >>>>> suggested
> > >>>>>>>>>> value]
> > >>>>>>>>>>
> > >>>>>>>>>> Current settings:
> > >>>>>>>>>> [DefaultMemoryPolicySize = value]
> > >>>>>>>>>> [{policy_name_1} size = value]
> > >>>>>>>>>> [{policy_name_1} size = value]
> > >>>>>>>>>> SystemCacheInitialSize = value
> > >>>>>>>>>> SystemCacheMaxSize = value
> > >>>>>>>>>> Java Heap Init Size = value
> > >>>>>>>>>> Java Heap Max Size = value
> > >>>>>>>>>>
> > >>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
> > >>> value
> > >>>>>>>>>> -------------------------------------------
> > >>>>>>>>>>
> > >>>>>>>>>> Records in […] are optional. If custom memory policy is not
> > >>> set
> > >>>> or
> > >>>>>> the
> > >>>>>>>>>> default memory policy is overridden the output will miss
> > >>> some of
> > >>>>> the
> > >>>>>>>> rows.
> > >>>>>>>>>>
> > >>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
> > >>>> parameter
> > >>>>>> was
> > >>>>>>>> set
> > >>>>>>>>>> explicitly by user code. Otherwise, the platform should be
> > >>> wise
> > >>>>>> enough
> > >>>>>>>> to
> > >>>>>>>>>> instantiate it properly depending on the host memory usage.
> > >>>>>>>>>>
> > >>>>>>>>>> —
> > >>>>>>>>>> Denis
> > >>>>>>>>>>
> > >>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> > >>>>>> gvvinblade@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> The message without logging layout:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Not enough memory for current process [required=29251MB,
> > >>>>>>>>>> available=15942MB].
> > >>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
> > >>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> > >>> memory
> > >>>>>>>> allocated
> > >>>>>>>>>>> for each node.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Current settings:
> > >>>>>>>>>>> HeapInit=250MB
> > >>>>>>>>>>> HeapMax=3543MB
> > >>>>>>>>>>> DefaultMemoryPolicySize=12753MB
> > >>>>>>>>>>> SystemCacheInitialSize=40MB
> > >>>>>>>>>>> SystemCacheMaxSize=100MB
> > >>>>>>>>>>>
> > >>>>>>>>>>> Other ignite instanses on the server require: 12853MB
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think it's make sense to describe what these numbers are
> > >>>>> consist
> > >>>>>>> of.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We simple say which parameters have an impact on how much
> > >>>> memory
> > >>>>>> the
> > >>>>>>>>>>> instance needs and their (parameters) actual values.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
> > >>>> the
> > >>>>>>> server
> > >>>>>>>>>> or
> > >>>>>>>>>>> workstation and it also consumes memory.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > >>>>>>>>>>> dsetrakyan@apache.org> написал:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
> > >>>>> text?
> > >>>>>>> Are
> > >>>>>>>> we
> > >>>>>>>>>>>> really going to print this whole thing out?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> > >>>>>>>> gvvinblade@gmail.com
> > >>>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> This message appears on topology change in case the
> > >>> available
> > >>>>>>> memory
> > >>>>>>>> is
> > >>>>>>>>>>>>> exceeded
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> > >>>>>> gvvinblade@gmail.com
> > >>>>>>>> :
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> An example of current impl:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
> > >>>>> memory
> > >>>>>>> for
> > >>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> > >>> change
> > >>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
> > >>>>> MemoryConfiguration.
> > >>>>>>>>>>>>> defaultMemoryPolicySize
> > >>>>>>>>>>>>>> to decrease memory allocated for each node.
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> > >>>>> settings:
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>> HeapInit=250MB
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>> HeapMax=3543MB
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>>>>>>>> DefaultMemoryPolicySize=
> > >>>>>>>>>>>>>> 12753MB
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>>>>>>>> SystemCacheMaxSize=100MB
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> > >>> ignite
> > >>>>>>>> instanses
> > >>>>>>>>>>>> on
> > >>>>>>>>>>>>>> the server require: 12853MB
> > >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > >>>>>> internal.
> > >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> > >>>>> skozlov@gridgain.com
> > >>>>>>> :
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
> > >>> suggest to
> > >>>>>>> reduce
> > >>>>>>>>>>>> both
> > >>>>>>>>>>>>>>> options
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > >>>>>>>>>>>>> dsetrakyan@apache.org
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
> > >>>>>> available
> > >>>>>>>> for
> > >>>>>>>>>>>>> OS.
> > >>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> > >>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
> > >>>> memory
> > >>>>>>>>>>>>> allocated
> > >>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>> each node.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
> > >>> for
> > >>>>> each
> > >>>>>>>> node
> > >>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>> be? In that case we should suggest it.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> D.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > >>>>>>>>>>>>> gvvinblade@gmail.com
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> What do you, guys think about next warning?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >>>>> GridHomePat
> > >>>>>>>>>>>>>>> hSelfTest0]
> > >>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
> > >>> MB.
> > >>>>> Use
> > >>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
> > >>>>> change
> > >>>>>>> the
> > >>>>>>>>>>>>>>>> setting.
> > >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >>>>> GridHomePat
> > >>>>>>>>>>>>>>> hSelfTest0]
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> > >>> memory
> > >>>>>>>>>>>> available
> > >>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>> OS.
> > >>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
> > >>>> usage.
> > >>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
> > >>> MemoryPolicyConfiguration
> > >>>> to
> > >>>>>>>>>>>> change
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> settings.
> > >>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
> > >>> available=15942MB]
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >>>>> GridHomePat
> > >>>>>>>>>>>>>>> hSelfTest0]
> > >>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
> > >>> production
> > >>>> for
> > >>>>>>>>>>>>>>> performance
> > >>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> deployment consistency reasons)
> > >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >>>>> GridHomePat
> > >>>>>>>>>>>>>>> hSelfTest0]
> > >>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> > >>>>>>>>>>>>> ['ignite-sys-cache']]
> > >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > >>>>> GridHomePat
> > >>>>>>>>>>>>>>> hSelfTest0]
> > >>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
> > >>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
> > >>> te/incubator-ignite/libs/
> > >>>>>>> licenses
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> > >>>>>> yzhdanov@apache.org
> > >>>>>>>> :
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
> > >>>>> mentioned
> > >>>>>>> by
> > >>>>>>>>>>>>>>> Denis
> > >>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>> Ivan.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> --Yakov
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>> Sergey Kozlov
> > >>>>>>>>>>>>>>> GridGain Systems
> > >>>>>>>>>>>>>>> www.gridgain.com
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >>
> >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Denis Magda <dm...@apache.org>.
Your suggestions sounds reasonable to me. Are they only about the calculations at the code level or you propose to change the message format too?
—
Denis
> On Aug 30, 2017, at 3:24 AM, Vladimir Ozerov <vo...@gridgain.com> wrote:
>
> I looked at the code of IGNITE-6182 (memory policy suggestions), and I do
> not like it. Too complex. My proposal:
> 1) Do not take in count JVM heap. First, we do not need whether it will be
> really used or not. Second, we never checked it in heap-based Ignite 1.x,
> and never had a single compliant.
> 2) We need to sum up all memory policies and *checkpoint buffer size*. If
> it exceeds 80% of machine's RAM - print a warning
>
> On Mon, Aug 28, 2017 at 7:26 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Anton, thanks for stepping in! The tickets set looks complete for me!
>>
>> Let’s start the vote once the fixes are merged and tests pass.
>>
>> —
>> Denis
>>
>>> On Aug 28, 2017, at 7:19 AM, Anton Vinogradov <av...@gridgain.com>
>> wrote:
>>>
>>> Igniters,
>>>
>>> Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
>>> In this case all other issues with fixVersion = 2.2 should be moved to
>> 2.3.
>>>
>>> Currently, I see 835 issues with fixVersion = 2.2
>>>
>>> Seems we should have only 4 issues with fixVersion = 2.2:
>>> - Change default max memory size from 80% to 20% -
>>> https://issues.apache.org/jira/browse/IGNITE-6182
>>> - Detecting low memory on ignite node startup -
>>> https://issues.apache.org/jira/browse/IGNITE-6003
>>> - Backport of improvements to checkpoint algorithm -
>>> https://issues.apache.org/jira/browse/IGNITE-????
>>> - ML profile is missing in 2.1 binary release -
>>> https://issues.apache.org/jira/browse/IGNITE-6193
>>>
>>> Please correct me in case I've missed something.
>>>
>>> Ivan,
>>>> Let's include to the release tickets with optimizations of checkpointing
>>> algorithm:
>>>> https://issues.apache.org/jira/browse/IGNITE-6178
>>>> https://issues.apache.org/jira/browse/IGNITE-6033
>>>> https://issues.apache.org/jira/browse/IGNITE-5961
>>>> This will help users who are experiencing problems with slow
>> checkpoints.
>>>> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183
>> to
>>> soften "Ignite node crashed in the middle of checkpoint" message as per
>>> discussion <
>>> http://apache-ignite-developers.2346864.n4.nabble.
>> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>>>> .
>>>
>>> All of these issues should have fixVersion 2.3.
>>> Please create backport issue and link it to all necessary issues.
>>>
>>> P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
>>>
>>> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
>> avinogradov@gridgain.com>
>>> wrote:
>>>
>>>> Denis,
>>>>
>>>>> BTW, who is considered to be the release manager of this release?
>>>> I'll do it.
>>>>
>>>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gvvinblade@gmail.com
>>>
>>>> wrote:
>>>>
>>>>> Ok, the check happens at the node start time or on NODE_JOIN event
>>>>>
>>>>> in general it looks like:
>>>>>
>>>>> 1) calculate expected used memory = heap max + system cache max + all
>>>>> custom policies max + default policy size and put it into a node
>> attribute
>>>>>
>>>>> 2) get total physycal memory, calculate expected safe to be used memory
>>>>> amount (leave 4 gb min or 20% of available memory for OS)
>>>>>
>>>>> 3) if expected used memory + expected used memory of other nodes on the
>>>>> host > than safe to be used memory amount, start calculating
>> suggestions
>>>>>
>>>>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
>>>>> 100mb default polycy, if available memory is less we cannot suggest
>>>>> anything reasonable, print warning, stop calculation.
>>>>>
>>>>> 5) check heap size (shouldn't exceed 30% of available memory
>> (total_memory
>>>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
>> just
>>>>> calculated value or 512MB minimal)
>>>>>
>>>>> 6) check if system cache size changed, suggest default value if it's so
>>>>>
>>>>> 7) in case 100 mb * policies count < available memory, suggest using
>>>>> default policy with max size equals to remaining memory (available -
>> heap
>>>>> -
>>>>> system cache)
>>>>>
>>>>> 8) calculate new size for each memory policy ( it's user defined size *
>>>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>>>> remaining memory, devided by nodes number on the host or 100 mb
>> minimal)
>>>>>
>>>>> 9) print suggestions
>>>>>
>>>>>
>>>>>
>>>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>>>
>>>>>> Igor, can you please describe the algorithm with all the thresholds?
>>>>>>
>>>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
>> gvvinblade@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> The suggestion here is based on initial settings, and it's so because
>>>>>> there
>>>>>>> is no other nodes on the host in the example.
>>>>>>>
>>>>>>> The algorithm tries to preserve the original ratio of memory policies
>>>>>>> keeping numbers reasonable (for example after some thresshold it will
>>>>>>> suggest not to use several memory policies if there is not enough of
>>>>>> memory
>>>>>>> for all of them) and taking into consideration nodes count on the
>>>>> host,
>>>>>>> each jvm heap, needed memory for OS, etc
>>>>>>>
>>>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
>>> :
>>>>>>>
>>>>>>>> Looks good, but why in the example provided are we suggesting 8GB? 2
>>>>>>> nodes
>>>>>>>> with 8GB will completely exhaust the available memory. I would
>>>>> suggest
>>>>>> 6
>>>>>>> or
>>>>>>>> 7GB.
>>>>>>>>
>>>>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
>>>>> small.
>>>>>>>>
>>>>>>>> Can you please comment?
>>>>>>>>
>>>>>>>> D.
>>>>>>>>
>>>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>>>>>> gvvinblade@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> One more example of possible warning:
>>>>>>>>>
>>>>>>>>> -----------------------------------------------------
>>>>>>>>> Excessive memory usage by Ignite node process (performance may
>>>>> drop)
>>>>>>>>> [requested=44613MB, available=15942MB].
>>>>>>>>>
>>>>>>>>> Please tune the folowing settings as suggested:
>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>>>>>>>>>
>>>>>>>>> Current settings:
>>>>>>>>> Java Heap maxSize: 3543MB
>>>>>>>>> Java Heap initSize: 250MB
>>>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>>>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>>>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>>>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>>>>>>>>> The overall expected memory usage by all Ignite nodes on the
>>>>> host:
>>>>>>>>> 44613MB
>>>>>>>>> -----------------------------------------------------
>>>>>>>>>
>>>>>>>>> Your thoughts?
>>>>>>>>>
>>>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>>>>>>>>>
>>>>>>>>>> Guys,
>>>>>>>>>>
>>>>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
>>>>>>> rolled
>>>>>>>>>> out in this emergency release:
>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>>>>>>>>>>
>>>>>>>>>> Oleg, Yuri, please step in and handle the issue.
>>>>>>>>>>
>>>>>>>>>> BTW, who is considered to be the release manager of this
>>>>> release?
>>>>>>>>>>
>>>>>>>>>> —
>>>>>>>>>> Denis
>>>>>>>>>>
>>>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>>>>>>>> dsetrakyan@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I like the format proposed by Denis, very clear.
>>>>>>>>>>>
>>>>>>>>>>> However, I also do not understand why a user should change the
>>>>>> size
>>>>>>>> of
>>>>>>>>>> some
>>>>>>>>>>> system cache. How would a user ever know what value to put
>>>>> there?
>>>>>>>> This
>>>>>>>>>>> value should be configured by Ignite automatically.
>>>>>>>>>>>
>>>>>>>>>>> D.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>>>>> dmagda@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Igor,
>>>>>>>>>>>>
>>>>>>>>>>>> Let me suggest this format.
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>> Excessive memory usage by Ignite node process (performance
>>>>> may
>>>>>>> drop)
>>>>>>>>>>>> [requested=29251MB, available=15942MB]
>>>>>>>>>>>>
>>>>>>>>>>>> Please tune the following settings:
>>>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
>>>>>> value]
>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
>>>>>>> suggested
>>>>>>>>>>>> value]
>>>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
>>>>>>> suggested
>>>>>>>>>>>> value]
>>>>>>>>>>>>
>>>>>>>>>>>> Current settings:
>>>>>>>>>>>> [DefaultMemoryPolicySize = value]
>>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>>>> SystemCacheInitialSize = value
>>>>>>>>>>>> SystemCacheMaxSize = value
>>>>>>>>>>>> Java Heap Init Size = value
>>>>>>>>>>>> Java Heap Max Size = value
>>>>>>>>>>>>
>>>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>>>>> value
>>>>>>>>>>>> -------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> Records in […] are optional. If custom memory policy is not
>>>>> set
>>>>>> or
>>>>>>>> the
>>>>>>>>>>>> default memory policy is overridden the output will miss
>>>>> some of
>>>>>>> the
>>>>>>>>>> rows.
>>>>>>>>>>>>
>>>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>>>>>> parameter
>>>>>>>> was
>>>>>>>>>> set
>>>>>>>>>>>> explicitly by user code. Otherwise, the platform should be
>>>>> wise
>>>>>>>> enough
>>>>>>>>>> to
>>>>>>>>>>>> instantiate it properly depending on the host memory usage.
>>>>>>>>>>>>
>>>>>>>>>>>> —
>>>>>>>>>>>> Denis
>>>>>>>>>>>>
>>>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>>>>>>>> gvvinblade@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The message without logging layout:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Not enough memory for current process [required=29251MB,
>>>>>>>>>>>> available=15942MB].
>>>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>>>>> memory
>>>>>>>>>> allocated
>>>>>>>>>>>>> for each node.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Current settings:
>>>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think it's make sense to describe what these numbers are
>>>>>>> consist
>>>>>>>>> of.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We simple say which parameters have an impact on how much
>>>>>> memory
>>>>>>>> the
>>>>>>>>>>>>> instance needs and their (parameters) actual values.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
>>>>>> the
>>>>>>>>> server
>>>>>>>>>>>> or
>>>>>>>>>>>>> workstation and it also consumes memory.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>>>>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
>>>>>>> text?
>>>>>>>>> Are
>>>>>>>>>> we
>>>>>>>>>>>>>> really going to print this whole thing out?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This message appears on topology change in case the
>>>>> available
>>>>>>>>> memory
>>>>>>>>>> is
>>>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> An example of current impl:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
>>>>>>> memory
>>>>>>>>> for
>>>>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>>>>> change
>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
>>>>>>> MemoryConfiguration.
>>>>>>>>>>>>>>> defaultMemoryPolicySize
>>>>>>>>>>>>>>>> to decrease memory allocated for each node.
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
>>>>>>> settings:
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>> HeapInit=250MB
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>> HeapMax=3543MB
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> DefaultMemoryPolicySize=
>>>>>>>>>>>>>>>> 12753MB
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>>>>> ignite
>>>>>>>>>> instanses
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> the server require: 12853MB
>>>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>>>> internal.
>>>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>>>>>>> skozlov@gridgain.com
>>>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
>>>>> suggest to
>>>>>>>>> reduce
>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
>>>>>>>> available
>>>>>>>>>> for
>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>>>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
>>>>>> memory
>>>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> each node.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
>>>>> for
>>>>>>> each
>>>>>>>>>> node
>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>> be? In that case we should suggest it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
>>>>> MB.
>>>>>>> Use
>>>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
>>>>>>> change
>>>>>>>>> the
>>>>>>>>>>>>>>>>>> setting.
>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>>>> memory
>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
>>>>>> usage.
>>>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
>>>>> MemoryPolicyConfiguration
>>>>>> to
>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> settings.
>>>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>>>>> available=15942MB]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>>>>> production
>>>>>> for
>>>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> deployment consistency reasons)
>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>>>>>>>>>>>> ['ignite-sys-cache']]
>>>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>>>> GridHomePat
>>>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>>>>> te/incubator-ignite/libs/
>>>>>>>>> licenses
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>>>>>>>> yzhdanov@apache.org
>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
>>>>>>> mentioned
>>>>>>>>> by
>>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> Ivan.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --Yakov
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Sergey Kozlov
>>>>>>>>>>>>>>>>> GridGain Systems
>>>>>>>>>>>>>>>>> www.gridgain.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Vladimir Ozerov <vo...@gridgain.com>.
I looked at the code of IGNITE-6182 (memory policy suggestions), and I do
not like it. Too complex. My proposal:
1) Do not take in count JVM heap. First, we do not need whether it will be
really used or not. Second, we never checked it in heap-based Ignite 1.x,
and never had a single compliant.
2) We need to sum up all memory policies and *checkpoint buffer size*. If
it exceeds 80% of machine's RAM - print a warning
On Mon, Aug 28, 2017 at 7:26 PM, Denis Magda <dm...@apache.org> wrote:
> Anton, thanks for stepping in! The tickets set looks complete for me!
>
> Let’s start the vote once the fixes are merged and tests pass.
>
> —
> Denis
>
> > On Aug 28, 2017, at 7:19 AM, Anton Vinogradov <av...@gridgain.com>
> wrote:
> >
> > Igniters,
> >
> > Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> > In this case all other issues with fixVersion = 2.2 should be moved to
> 2.3.
> >
> > Currently, I see 835 issues with fixVersion = 2.2
> >
> > Seems we should have only 4 issues with fixVersion = 2.2:
> > - Change default max memory size from 80% to 20% -
> > https://issues.apache.org/jira/browse/IGNITE-6182
> > - Detecting low memory on ignite node startup -
> > https://issues.apache.org/jira/browse/IGNITE-6003
> > - Backport of improvements to checkpoint algorithm -
> > https://issues.apache.org/jira/browse/IGNITE-????
> > - ML profile is missing in 2.1 binary release -
> > https://issues.apache.org/jira/browse/IGNITE-6193
> >
> > Please correct me in case I've missed something.
> >
> > Ivan,
> >> Let's include to the release tickets with optimizations of checkpointing
> > algorithm:
> >> https://issues.apache.org/jira/browse/IGNITE-6178
> >> https://issues.apache.org/jira/browse/IGNITE-6033
> >> https://issues.apache.org/jira/browse/IGNITE-5961
> >> This will help users who are experiencing problems with slow
> checkpoints.
> >> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183
> to
> > soften "Ignite node crashed in the middle of checkpoint" message as per
> > discussion <
> > http://apache-ignite-developers.2346864.n4.nabble.
> com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
> >> .
> >
> > All of these issues should have fixVersion 2.3.
> > Please create backport issue and link it to all necessary issues.
> >
> > P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
> >
> > On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <
> avinogradov@gridgain.com>
> > wrote:
> >
> >> Denis,
> >>
> >>> BTW, who is considered to be the release manager of this release?
> >> I'll do it.
> >>
> >> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gvvinblade@gmail.com
> >
> >> wrote:
> >>
> >>> Ok, the check happens at the node start time or on NODE_JOIN event
> >>>
> >>> in general it looks like:
> >>>
> >>> 1) calculate expected used memory = heap max + system cache max + all
> >>> custom policies max + default policy size and put it into a node
> attribute
> >>>
> >>> 2) get total physycal memory, calculate expected safe to be used memory
> >>> amount (leave 4 gb min or 20% of available memory for OS)
> >>>
> >>> 3) if expected used memory + expected used memory of other nodes on the
> >>> host > than safe to be used memory amount, start calculating
> suggestions
> >>>
> >>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
> >>> 100mb default polycy, if available memory is less we cannot suggest
> >>> anything reasonable, print warning, stop calculation.
> >>>
> >>> 5) check heap size (shouldn't exceed 30% of available memory
> (total_memory
> >>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest
> just
> >>> calculated value or 512MB minimal)
> >>>
> >>> 6) check if system cache size changed, suggest default value if it's so
> >>>
> >>> 7) in case 100 mb * policies count < available memory, suggest using
> >>> default policy with max size equals to remaining memory (available -
> heap
> >>> -
> >>> system cache)
> >>>
> >>> 8) calculate new size for each memory policy ( it's user defined size *
> >>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> >>> remaining memory, devided by nodes number on the host or 100 mb
> minimal)
> >>>
> >>> 9) print suggestions
> >>>
> >>>
> >>>
> >>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> >>>
> >>>> Igor, can you please describe the algorithm with all the thresholds?
> >>>>
> >>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <
> gvvinblade@gmail.com
> >>>>
> >>>> wrote:
> >>>>
> >>>>> The suggestion here is based on initial settings, and it's so because
> >>>> there
> >>>>> is no other nodes on the host in the example.
> >>>>>
> >>>>> The algorithm tries to preserve the original ratio of memory policies
> >>>>> keeping numbers reasonable (for example after some thresshold it will
> >>>>> suggest not to use several memory policies if there is not enough of
> >>>> memory
> >>>>> for all of them) and taking into consideration nodes count on the
> >>> host,
> >>>>> each jvm heap, needed memory for OS, etc
> >>>>>
> >>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org
> >:
> >>>>>
> >>>>>> Looks good, but why in the example provided are we suggesting 8GB? 2
> >>>>> nodes
> >>>>>> with 8GB will completely exhaust the available memory. I would
> >>> suggest
> >>>> 6
> >>>>> or
> >>>>>> 7GB.
> >>>>>>
> >>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
> >>> small.
> >>>>>>
> >>>>>> Can you please comment?
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> >>>> gvvinblade@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> One more example of possible warning:
> >>>>>>>
> >>>>>>> -----------------------------------------------------
> >>>>>>> Excessive memory usage by Ignite node process (performance may
> >>> drop)
> >>>>>>> [requested=44613MB, available=15942MB].
> >>>>>>>
> >>>>>>> Please tune the folowing settings as suggested:
> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> >>>>>>>
> >>>>>>> Current settings:
> >>>>>>> Java Heap maxSize: 3543MB
> >>>>>>> Java Heap initSize: 250MB
> >>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> >>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> >>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> >>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> >>>>>>> The overall expected memory usage by all Ignite nodes on the
> >>> host:
> >>>>>>> 44613MB
> >>>>>>> -----------------------------------------------------
> >>>>>>>
> >>>>>>> Your thoughts?
> >>>>>>>
> >>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> >>>>>>>
> >>>>>>>> Guys,
> >>>>>>>>
> >>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
> >>>>> rolled
> >>>>>>>> out in this emergency release:
> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
> >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
> >>>>>>>>
> >>>>>>>> Oleg, Yuri, please step in and handle the issue.
> >>>>>>>>
> >>>>>>>> BTW, who is considered to be the release manager of this
> >>> release?
> >>>>>>>>
> >>>>>>>> —
> >>>>>>>> Denis
> >>>>>>>>
> >>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> >>>>>> dsetrakyan@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> I like the format proposed by Denis, very clear.
> >>>>>>>>>
> >>>>>>>>> However, I also do not understand why a user should change the
> >>>> size
> >>>>>> of
> >>>>>>>> some
> >>>>>>>>> system cache. How would a user ever know what value to put
> >>> there?
> >>>>>> This
> >>>>>>>>> value should be configured by Ignite automatically.
> >>>>>>>>>
> >>>>>>>>> D.
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> >>> dmagda@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Igor,
> >>>>>>>>>>
> >>>>>>>>>> Let me suggest this format.
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------
> >>>>>>>>>> Excessive memory usage by Ignite node process (performance
> >>> may
> >>>>> drop)
> >>>>>>>>>> [requested=29251MB, available=15942MB]
> >>>>>>>>>>
> >>>>>>>>>> Please tune the following settings:
> >>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
> >>>> value]
> >>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> >>>>> suggested
> >>>>>>>>>> value]
> >>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> >>>>> suggested
> >>>>>>>>>> value]
> >>>>>>>>>>
> >>>>>>>>>> Current settings:
> >>>>>>>>>> [DefaultMemoryPolicySize = value]
> >>>>>>>>>> [{policy_name_1} size = value]
> >>>>>>>>>> [{policy_name_1} size = value]
> >>>>>>>>>> SystemCacheInitialSize = value
> >>>>>>>>>> SystemCacheMaxSize = value
> >>>>>>>>>> Java Heap Init Size = value
> >>>>>>>>>> Java Heap Max Size = value
> >>>>>>>>>>
> >>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
> >>> value
> >>>>>>>>>> -------------------------------------------
> >>>>>>>>>>
> >>>>>>>>>> Records in […] are optional. If custom memory policy is not
> >>> set
> >>>> or
> >>>>>> the
> >>>>>>>>>> default memory policy is overridden the output will miss
> >>> some of
> >>>>> the
> >>>>>>>> rows.
> >>>>>>>>>>
> >>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
> >>>> parameter
> >>>>>> was
> >>>>>>>> set
> >>>>>>>>>> explicitly by user code. Otherwise, the platform should be
> >>> wise
> >>>>>> enough
> >>>>>>>> to
> >>>>>>>>>> instantiate it properly depending on the host memory usage.
> >>>>>>>>>>
> >>>>>>>>>> —
> >>>>>>>>>> Denis
> >>>>>>>>>>
> >>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> >>>>>> gvvinblade@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> The message without logging layout:
> >>>>>>>>>>>
> >>>>>>>>>>> Not enough memory for current process [required=29251MB,
> >>>>>>>>>> available=15942MB].
> >>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
> >>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> >>> memory
> >>>>>>>> allocated
> >>>>>>>>>>> for each node.
> >>>>>>>>>>>
> >>>>>>>>>>> Current settings:
> >>>>>>>>>>> HeapInit=250MB
> >>>>>>>>>>> HeapMax=3543MB
> >>>>>>>>>>> DefaultMemoryPolicySize=12753MB
> >>>>>>>>>>> SystemCacheInitialSize=40MB
> >>>>>>>>>>> SystemCacheMaxSize=100MB
> >>>>>>>>>>>
> >>>>>>>>>>> Other ignite instanses on the server require: 12853MB
> >>>>>>>>>>>
> >>>>>>>>>>> I think it's make sense to describe what these numbers are
> >>>>> consist
> >>>>>>> of.
> >>>>>>>>>>>
> >>>>>>>>>>> We simple say which parameters have an impact on how much
> >>>> memory
> >>>>>> the
> >>>>>>>>>>> instance needs and their (parameters) actual values.
> >>>>>>>>>>>
> >>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
> >>>> the
> >>>>>>> server
> >>>>>>>>>> or
> >>>>>>>>>>> workstation and it also consumes memory.
> >>>>>>>>>>>
> >>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> >>>>>>>>>>> dsetrakyan@apache.org> написал:
> >>>>>>>>>>>
> >>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
> >>>>> text?
> >>>>>>> Are
> >>>>>>>> we
> >>>>>>>>>>>> really going to print this whole thing out?
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> >>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> This message appears on topology change in case the
> >>> available
> >>>>>>> memory
> >>>>>>>> is
> >>>>>>>>>>>>> exceeded
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> >>>>>> gvvinblade@gmail.com
> >>>>>>>> :
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> An example of current impl:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
> >>>>> memory
> >>>>>>> for
> >>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> >>> change
> >>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
> >>>>> MemoryConfiguration.
> >>>>>>>>>>>>> defaultMemoryPolicySize
> >>>>>>>>>>>>>> to decrease memory allocated for each node.
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> >>>>> settings:
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>> HeapInit=250MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>> HeapMax=3543MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>> DefaultMemoryPolicySize=
> >>>>>>>>>>>>>> 12753MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>> SystemCacheInitialSize=40MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>> SystemCacheMaxSize=100MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> >>> ignite
> >>>>>>>> instanses
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>> the server require: 12853MB
> >>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> >>>>>> internal.
> >>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> >>>>> skozlov@gridgain.com
> >>>>>>> :
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
> >>> suggest to
> >>>>>>> reduce
> >>>>>>>>>>>> both
> >>>>>>>>>>>>>>> options
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> >>>>>>>>>>>>> dsetrakyan@apache.org
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
> >>>>>> available
> >>>>>>>> for
> >>>>>>>>>>>>> OS.
> >>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> >>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
> >>>> memory
> >>>>>>>>>>>>> allocated
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> each node.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
> >>> for
> >>>>> each
> >>>>>>>> node
> >>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>> be? In that case we should suggest it.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> >>>>>>>>>>>>> gvvinblade@gmail.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> What do you, guys think about next warning?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
> >>> MB.
> >>>>> Use
> >>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
> >>>>> change
> >>>>>>> the
> >>>>>>>>>>>>>>>> setting.
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
> >>> memory
> >>>>>>>>>>>> available
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> OS.
> >>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
> >>>> usage.
> >>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
> >>> MemoryPolicyConfiguration
> >>>> to
> >>>>>>>>>>>> change
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> settings.
> >>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
> >>> available=15942MB]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
> >>> production
> >>>> for
> >>>>>>>>>>>>>>> performance
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> deployment consistency reasons)
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> >>>>>>>>>>>>> ['ignite-sys-cache']]
> >>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
> >>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> >>>>> GridHomePat
> >>>>>>>>>>>>>>> hSelfTest0]
> >>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
> >>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
> >>> te/incubator-ignite/libs/
> >>>>>>> licenses
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> >>>>>> yzhdanov@apache.org
> >>>>>>>> :
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
> >>>>> mentioned
> >>>>>>> by
> >>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> Ivan.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --Yakov
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Sergey Kozlov
> >>>>>>>>>>>>>>> GridGain Systems
> >>>>>>>>>>>>>>> www.gridgain.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Denis Magda <dm...@apache.org>.
Anton, thanks for stepping in! The tickets set looks complete for me!
Let’s start the vote once the fixes are merged and tests pass.
—
Denis
> On Aug 28, 2017, at 7:19 AM, Anton Vinogradov <av...@gridgain.com> wrote:
>
> Igniters,
>
> Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> In this case all other issues with fixVersion = 2.2 should be moved to 2.3.
>
> Currently, I see 835 issues with fixVersion = 2.2
>
> Seems we should have only 4 issues with fixVersion = 2.2:
> - Change default max memory size from 80% to 20% -
> https://issues.apache.org/jira/browse/IGNITE-6182
> - Detecting low memory on ignite node startup -
> https://issues.apache.org/jira/browse/IGNITE-6003
> - Backport of improvements to checkpoint algorithm -
> https://issues.apache.org/jira/browse/IGNITE-????
> - ML profile is missing in 2.1 binary release -
> https://issues.apache.org/jira/browse/IGNITE-6193
>
> Please correct me in case I've missed something.
>
> Ivan,
>> Let's include to the release tickets with optimizations of checkpointing
> algorithm:
>> https://issues.apache.org/jira/browse/IGNITE-6178
>> https://issues.apache.org/jira/browse/IGNITE-6033
>> https://issues.apache.org/jira/browse/IGNITE-5961
>> This will help users who are experiencing problems with slow checkpoints.
>> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to
> soften "Ignite node crashed in the middle of checkpoint" message as per
> discussion <
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>> .
>
> All of these issues should have fixVersion 2.3.
> Please create backport issue and link it to all necessary issues.
>
> P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
>
> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <av...@gridgain.com>
> wrote:
>
>> Denis,
>>
>>> BTW, who is considered to be the release manager of this release?
>> I'll do it.
>>
>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gv...@gmail.com>
>> wrote:
>>
>>> Ok, the check happens at the node start time or on NODE_JOIN event
>>>
>>> in general it looks like:
>>>
>>> 1) calculate expected used memory = heap max + system cache max + all
>>> custom policies max + default policy size and put it into a node attribute
>>>
>>> 2) get total physycal memory, calculate expected safe to be used memory
>>> amount (leave 4 gb min or 20% of available memory for OS)
>>>
>>> 3) if expected used memory + expected used memory of other nodes on the
>>> host > than safe to be used memory amount, start calculating suggestions
>>>
>>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
>>> 100mb default polycy, if available memory is less we cannot suggest
>>> anything reasonable, print warning, stop calculation.
>>>
>>> 5) check heap size (shouldn't exceed 30% of available memory (total_memory
>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
>>> calculated value or 512MB minimal)
>>>
>>> 6) check if system cache size changed, suggest default value if it's so
>>>
>>> 7) in case 100 mb * policies count < available memory, suggest using
>>> default policy with max size equals to remaining memory (available - heap
>>> -
>>> system cache)
>>>
>>> 8) calculate new size for each memory policy ( it's user defined size *
>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>> remaining memory, devided by nodes number on the host or 100 mb minimal)
>>>
>>> 9) print suggestions
>>>
>>>
>>>
>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>
>>>> Igor, can you please describe the algorithm with all the thresholds?
>>>>
>>>> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gvvinblade@gmail.com
>>>>
>>>> wrote:
>>>>
>>>>> The suggestion here is based on initial settings, and it's so because
>>>> there
>>>>> is no other nodes on the host in the example.
>>>>>
>>>>> The algorithm tries to preserve the original ratio of memory policies
>>>>> keeping numbers reasonable (for example after some thresshold it will
>>>>> suggest not to use several memory policies if there is not enough of
>>>> memory
>>>>> for all of them) and taking into consideration nodes count on the
>>> host,
>>>>> each jvm heap, needed memory for OS, etc
>>>>>
>>>>> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>>>
>>>>>> Looks good, but why in the example provided are we suggesting 8GB? 2
>>>>> nodes
>>>>>> with 8GB will completely exhaust the available memory. I would
>>> suggest
>>>> 6
>>>>> or
>>>>>> 7GB.
>>>>>>
>>>>>> Also, why 100MB for default policy. Anything under 1GB seems too
>>> small.
>>>>>>
>>>>>> Can you please comment?
>>>>>>
>>>>>> D.
>>>>>>
>>>>>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>>>> gvvinblade@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> One more example of possible warning:
>>>>>>>
>>>>>>> -----------------------------------------------------
>>>>>>> Excessive memory usage by Ignite node process (performance may
>>> drop)
>>>>>>> [requested=44613MB, available=15942MB].
>>>>>>>
>>>>>>> Please tune the folowing settings as suggested:
>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>>>>>>>
>>>>>>> Current settings:
>>>>>>> Java Heap maxSize: 3543MB
>>>>>>> Java Heap initSize: 250MB
>>>>>>> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>>>>>>> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>>>>>>> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>>>>>>> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>>>>>>> The overall expected memory usage by all Ignite nodes on the
>>> host:
>>>>>>> 44613MB
>>>>>>> -----------------------------------------------------
>>>>>>>
>>>>>>> Your thoughts?
>>>>>>>
>>>>>>> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>>>>>>>
>>>>>>>> Guys,
>>>>>>>>
>>>>>>>> ML lib profile is missing in 2.1 release! That must be fixed and
>>>>> rolled
>>>>>>>> out in this emergency release:
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193 <
>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-6193>
>>>>>>>>
>>>>>>>> Oleg, Yuri, please step in and handle the issue.
>>>>>>>>
>>>>>>>> BTW, who is considered to be the release manager of this
>>> release?
>>>>>>>>
>>>>>>>> —
>>>>>>>> Denis
>>>>>>>>
>>>>>>>>> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>>>>>> dsetrakyan@apache.org>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I like the format proposed by Denis, very clear.
>>>>>>>>>
>>>>>>>>> However, I also do not understand why a user should change the
>>>> size
>>>>>> of
>>>>>>>> some
>>>>>>>>> system cache. How would a user ever know what value to put
>>> there?
>>>>>> This
>>>>>>>>> value should be configured by Ignite automatically.
>>>>>>>>>
>>>>>>>>> D.
>>>>>>>>>
>>>>>>>>> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>>> dmagda@apache.org>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Igor,
>>>>>>>>>>
>>>>>>>>>> Let me suggest this format.
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------
>>>>>>>>>> Excessive memory usage by Ignite node process (performance
>>> may
>>>>> drop)
>>>>>>>>>> [requested=29251MB, available=15942MB]
>>>>>>>>>>
>>>>>>>>>> Please tune the following settings:
>>>>>>>>>> [MemoryConfiguration.defaultMemoryPolicySize = suggested
>>>> value]
>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize = suggested value
>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
>>>>> suggested
>>>>>>>>>> value]
>>>>>>>>>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
>>>>> suggested
>>>>>>>>>> value]
>>>>>>>>>>
>>>>>>>>>> Current settings:
>>>>>>>>>> [DefaultMemoryPolicySize = value]
>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>> [{policy_name_1} size = value]
>>>>>>>>>> SystemCacheInitialSize = value
>>>>>>>>>> SystemCacheMaxSize = value
>>>>>>>>>> Java Heap Init Size = value
>>>>>>>>>> Java Heap Max Size = value
>>>>>>>>>>
>>>>>>>>>> The overall memory usage by all Ignite nodes on the host:
>>> value
>>>>>>>>>> -------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Records in […] are optional. If custom memory policy is not
>>> set
>>>> or
>>>>>> the
>>>>>>>>>> default memory policy is overridden the output will miss
>>> some of
>>>>> the
>>>>>>>> rows.
>>>>>>>>>>
>>>>>>>>>> As for systemCacheMaxSize, it should be show ONLY if the
>>>> parameter
>>>>>> was
>>>>>>>> set
>>>>>>>>>> explicitly by user code. Otherwise, the platform should be
>>> wise
>>>>>> enough
>>>>>>>> to
>>>>>>>>>> instantiate it properly depending on the host memory usage.
>>>>>>>>>>
>>>>>>>>>> —
>>>>>>>>>> Denis
>>>>>>>>>>
>>>>>>>>>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>>>>>> gvvinblade@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> The message without logging layout:
>>>>>>>>>>>
>>>>>>>>>>> Not enough memory for current process [required=29251MB,
>>>>>>>>>> available=15942MB].
>>>>>>>>>>> Please change MemoryConfiguration.systemCacheMaxSize and
>>>>>>>>>>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>>> memory
>>>>>>>> allocated
>>>>>>>>>>> for each node.
>>>>>>>>>>>
>>>>>>>>>>> Current settings:
>>>>>>>>>>> HeapInit=250MB
>>>>>>>>>>> HeapMax=3543MB
>>>>>>>>>>> DefaultMemoryPolicySize=12753MB
>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>
>>>>>>>>>>> Other ignite instanses on the server require: 12853MB
>>>>>>>>>>>
>>>>>>>>>>> I think it's make sense to describe what these numbers are
>>>>> consist
>>>>>>> of.
>>>>>>>>>>>
>>>>>>>>>>> We simple say which parameters have an impact on how much
>>>> memory
>>>>>> the
>>>>>>>>>>> instance needs and their (parameters) actual values.
>>>>>>>>>>>
>>>>>>>>>>> Also we notice that more than one Ignite instance are ran on
>>>> the
>>>>>>> server
>>>>>>>>>> or
>>>>>>>>>>> workstation and it also consumes memory.
>>>>>>>>>>>
>>>>>>>>>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>>>>
>>>>>>>>>>>> Igor, what is this flood of WARN messaging coming after the
>>>>> text?
>>>>>>> Are
>>>>>>>> we
>>>>>>>>>>>> really going to print this whole thing out?
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> This message appears on topology change in case the
>>> available
>>>>>>> memory
>>>>>>>> is
>>>>>>>>>>>>> exceeded
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>>>>>> gvvinblade@gmail.com
>>>>>>>> :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> An example of current impl:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
>>>>> memory
>>>>>>> for
>>>>>>>>>>>>>> current process [required=29251MB, available=15942MB].
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>>> change
>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMaxSize and
>>>>> MemoryConfiguration.
>>>>>>>>>>>>> defaultMemoryPolicySize
>>>>>>>>>>>>>> to decrease memory allocated for each node.
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
>>>>> settings:
>>>>>>>>>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>> HeapInit=250MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>> HeapMax=3543MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>> DefaultMemoryPolicySize=
>>>>>>>>>>>>>> 12753MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> SystemCacheInitialSize=40MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>> SystemCacheMaxSize=100MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>>> ignite
>>>>>>>> instanses
>>>>>>>>>>>> on
>>>>>>>>>>>>>> the server require: 12853MB
>>>>>>>>>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>>>>>> internal.
>>>>>>>>>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>>>>> skozlov@gridgain.com
>>>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I suppose we should not forget JVM heap size and
>>> suggest to
>>>>>>> reduce
>>>>>>>>>>>> both
>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Required RAM size is larger than total physical memory
>>>>>> available
>>>>>>>> for
>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>>>>>>>>>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
>>>> memory
>>>>>>>>>>>>> allocated
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> each node.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, can we calculate what the memory size allocated
>>> for
>>>>> each
>>>>>>>> node
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> be? In that case we should suggest it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>>>>>>>>>> gvvinblade@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> System cache's MemoryPolicy size is configured to 40
>>> MB.
>>>>> Use
>>>>>>>>>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
>>>>> change
>>>>>>> the
>>>>>>>>>>>>>>>> setting.
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Required RAM size is larger than total physical
>>> memory
>>>>>>>>>>>> available
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> OS.
>>>>>>>>>>>>>>>>>>>> Check your configuration to avoid swap partition
>>>> usage.
>>>>>>>>>>>>>>>>>>>> Use MemoryConfiguration and
>>> MemoryPolicyConfiguration
>>>> to
>>>>>>>>>>>> change
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> settings.
>>>>>>>>>>>>>>>>>>>> Physical memory [required=16397MB,
>>> available=15942MB]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> Peer class loading is enabled (disable it in
>>> production
>>>> for
>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> deployment consistency reasons)
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>>>>>>>>>> ['ignite-sys-cache']]
>>>>>>>>>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>>>>> GridHomePat
>>>>>>>>>>>>>>> hSelfTest0]
>>>>>>>>>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>>>>>>>>>> /home/gvvinblade/projects/igni
>>> te/incubator-ignite/libs/
>>>>>>> licenses
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>>>>>> yzhdanov@apache.org
>>>>>>>> :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Agree, let's release new version including tickets
>>>>> mentioned
>>>>>>> by
>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> Ivan.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --Yakov
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Sergey Kozlov
>>>>>>>>>>>>>>> GridGain Systems
>>>>>>>>>>>>>>> www.gridgain.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Anton Vinogradov <av...@gridgain.com>.
Igniters,
Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
In this case all other issues with fixVersion = 2.2 should be moved to 2.3.
Currently, I see 835 issues with fixVersion = 2.2
Seems we should have only 4 issues with fixVersion = 2.2:
- Change default max memory size from 80% to 20% -
https://issues.apache.org/jira/browse/IGNITE-6182
- Detecting low memory on ignite node startup -
https://issues.apache.org/jira/browse/IGNITE-6003
- Backport of improvements to checkpoint algorithm -
https://issues.apache.org/jira/browse/IGNITE-????
- ML profile is missing in 2.1 binary release -
https://issues.apache.org/jira/browse/IGNITE-6193
Please correct me in case I've missed something.
Ivan,
> Let's include to the release tickets with optimizations of checkpointing
algorithm:
> https://issues.apache.org/jira/browse/IGNITE-6178
> https://issues.apache.org/jira/browse/IGNITE-6033
> https://issues.apache.org/jira/browse/IGNITE-5961
> This will help users who are experiencing problems with slow checkpoints.
> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to
soften "Ignite node crashed in the middle of checkpoint" message as per
discussion <
http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>.
All of these issues should have fixVersion 2.3.
Please create backport issue and link it to all necessary issues.
P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov <av...@gridgain.com>
wrote:
> Denis,
>
> > BTW, who is considered to be the release manager of this release?
> I'll do it.
>
> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gv...@gmail.com>
> wrote:
>
>> Ok, the check happens at the node start time or on NODE_JOIN event
>>
>> in general it looks like:
>>
>> 1) calculate expected used memory = heap max + system cache max + all
>> custom policies max + default policy size and put it into a node attribute
>>
>> 2) get total physycal memory, calculate expected safe to be used memory
>> amount (leave 4 gb min or 20% of available memory for OS)
>>
>> 3) if expected used memory + expected used memory of other nodes on the
>> host > than safe to be used memory amount, start calculating suggestions
>>
>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
>> 100mb default polycy, if available memory is less we cannot suggest
>> anything reasonable, print warning, stop calculation.
>>
>> 5) check heap size (shouldn't exceed 30% of available memory (total_memory
>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
>> calculated value or 512MB minimal)
>>
>> 6) check if system cache size changed, suggest default value if it's so
>>
>> 7) in case 100 mb * policies count < available memory, suggest using
>> default policy with max size equals to remaining memory (available - heap
>> -
>> system cache)
>>
>> 8) calculate new size for each memory policy ( it's user defined size *
>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>> remaining memory, devided by nodes number on the host or 100 mb minimal)
>>
>> 9) print suggestions
>>
>>
>>
>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>
>> > Igor, can you please describe the algorithm with all the thresholds?
>> >
>> > On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gvvinblade@gmail.com
>> >
>> > wrote:
>> >
>> > > The suggestion here is based on initial settings, and it's so because
>> > there
>> > > is no other nodes on the host in the example.
>> > >
>> > > The algorithm tries to preserve the original ratio of memory policies
>> > > keeping numbers reasonable (for example after some thresshold it will
>> > > suggest not to use several memory policies if there is not enough of
>> > memory
>> > > for all of them) and taking into consideration nodes count on the
>> host,
>> > > each jvm heap, needed memory for OS, etc
>> > >
>> > > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>> > >
>> > > > Looks good, but why in the example provided are we suggesting 8GB? 2
>> > > nodes
>> > > > with 8GB will completely exhaust the available memory. I would
>> suggest
>> > 6
>> > > or
>> > > > 7GB.
>> > > >
>> > > > Also, why 100MB for default policy. Anything under 1GB seems too
>> small.
>> > > >
>> > > > Can you please comment?
>> > > >
>> > > > D.
>> > > >
>> > > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>> > gvvinblade@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > One more example of possible warning:
>> > > > >
>> > > > > -----------------------------------------------------
>> > > > > Excessive memory usage by Ignite node process (performance may
>> drop)
>> > > > > [requested=44613MB, available=15942MB].
>> > > > >
>> > > > > Please tune the folowing settings as suggested:
>> > > > > MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>> > > > > MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>> > > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>> > > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>> > > > >
>> > > > > Current settings:
>> > > > > Java Heap maxSize: 3543MB
>> > > > > Java Heap initSize: 250MB
>> > > > > MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>> > > > > MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>> > > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>> > > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>> > > > > The overall expected memory usage by all Ignite nodes on the
>> host:
>> > > > > 44613MB
>> > > > > -----------------------------------------------------
>> > > > >
>> > > > > Your thoughts?
>> > > > >
>> > > > > 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>> > > > >
>> > > > > > Guys,
>> > > > > >
>> > > > > > ML lib profile is missing in 2.1 release! That must be fixed and
>> > > rolled
>> > > > > > out in this emergency release:
>> > > > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
>> > > > > > https://issues.apache.org/jira/browse/IGNITE-6193>
>> > > > > >
>> > > > > > Oleg, Yuri, please step in and handle the issue.
>> > > > > >
>> > > > > > BTW, who is considered to be the release manager of this
>> release?
>> > > > > >
>> > > > > > —
>> > > > > > Denis
>> > > > > >
>> > > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
>> > > > dsetrakyan@apache.org>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > I like the format proposed by Denis, very clear.
>> > > > > > >
>> > > > > > > However, I also do not understand why a user should change the
>> > size
>> > > > of
>> > > > > > some
>> > > > > > > system cache. How would a user ever know what value to put
>> there?
>> > > > This
>> > > > > > > value should be configured by Ignite automatically.
>> > > > > > >
>> > > > > > > D.
>> > > > > > >
>> > > > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
>> dmagda@apache.org>
>> > > > > wrote:
>> > > > > > >
>> > > > > > >> Igor,
>> > > > > > >>
>> > > > > > >> Let me suggest this format.
>> > > > > > >>
>> > > > > > >> ---------------------------------------------
>> > > > > > >> Excessive memory usage by Ignite node process (performance
>> may
>> > > drop)
>> > > > > > >> [requested=29251MB, available=15942MB]
>> > > > > > >>
>> > > > > > >> Please tune the following settings:
>> > > > > > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested
>> > value]
>> > > > > > >> MemoryConfiguration.systemCacheMaxSize = suggested value
>> > > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
>> > > suggested
>> > > > > > >> value]
>> > > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
>> > > suggested
>> > > > > > >> value]
>> > > > > > >>
>> > > > > > >> Current settings:
>> > > > > > >> [DefaultMemoryPolicySize = value]
>> > > > > > >> [{policy_name_1} size = value]
>> > > > > > >> [{policy_name_1} size = value]
>> > > > > > >> SystemCacheInitialSize = value
>> > > > > > >> SystemCacheMaxSize = value
>> > > > > > >> Java Heap Init Size = value
>> > > > > > >> Java Heap Max Size = value
>> > > > > > >>
>> > > > > > >> The overall memory usage by all Ignite nodes on the host:
>> value
>> > > > > > >> -------------------------------------------
>> > > > > > >>
>> > > > > > >> Records in […] are optional. If custom memory policy is not
>> set
>> > or
>> > > > the
>> > > > > > >> default memory policy is overridden the output will miss
>> some of
>> > > the
>> > > > > > rows.
>> > > > > > >>
>> > > > > > >> As for systemCacheMaxSize, it should be show ONLY if the
>> > parameter
>> > > > was
>> > > > > > set
>> > > > > > >> explicitly by user code. Otherwise, the platform should be
>> wise
>> > > > enough
>> > > > > > to
>> > > > > > >> instantiate it properly depending on the host memory usage.
>> > > > > > >>
>> > > > > > >> —
>> > > > > > >> Denis
>> > > > > > >>
>> > > > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
>> > > > gvvinblade@gmail.com>
>> > > > > > >> wrote:
>> > > > > > >>>
>> > > > > > >>> The message without logging layout:
>> > > > > > >>>
>> > > > > > >>> Not enough memory for current process [required=29251MB,
>> > > > > > >> available=15942MB].
>> > > > > > >>> Please change MemoryConfiguration.systemCacheMaxSize and
>> > > > > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease
>> memory
>> > > > > > allocated
>> > > > > > >>> for each node.
>> > > > > > >>>
>> > > > > > >>> Current settings:
>> > > > > > >>> HeapInit=250MB
>> > > > > > >>> HeapMax=3543MB
>> > > > > > >>> DefaultMemoryPolicySize=12753MB
>> > > > > > >>> SystemCacheInitialSize=40MB
>> > > > > > >>> SystemCacheMaxSize=100MB
>> > > > > > >>>
>> > > > > > >>> Other ignite instanses on the server require: 12853MB
>> > > > > > >>>
>> > > > > > >>> I think it's make sense to describe what these numbers are
>> > > consist
>> > > > > of.
>> > > > > > >>>
>> > > > > > >>> We simple say which parameters have an impact on how much
>> > memory
>> > > > the
>> > > > > > >>> instance needs and their (parameters) actual values.
>> > > > > > >>>
>> > > > > > >>> Also we notice that more than one Ignite instance are ran on
>> > the
>> > > > > server
>> > > > > > >> or
>> > > > > > >>> workstation and it also consumes memory.
>> > > > > > >>>
>> > > > > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>> > > > > > >>> dsetrakyan@apache.org> написал:
>> > > > > > >>>
>> > > > > > >>>> Igor, what is this flood of WARN messaging coming after the
>> > > text?
>> > > > > Are
>> > > > > > we
>> > > > > > >>>> really going to print this whole thing out?
>> > > > > > >>>>
>> > > > > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
>> > > > > > gvvinblade@gmail.com
>> > > > > > >>>
>> > > > > > >>>> wrote:
>> > > > > > >>>>
>> > > > > > >>>>> This message appears on topology change in case the
>> available
>> > > > > memory
>> > > > > > is
>> > > > > > >>>>> exceeded
>> > > > > > >>>>>
>> > > > > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
>> > > > gvvinblade@gmail.com
>> > > > > >:
>> > > > > > >>>>>
>> > > > > > >>>>>> An example of current impl:
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
>> > > memory
>> > > > > for
>> > > > > > >>>>>> current process [required=29251MB, available=15942MB].
>> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
>> change
>> > > > > > >>>>>> MemoryConfiguration.systemCacheMaxSize and
>> > > MemoryConfiguration.
>> > > > > > >>>>> defaultMemoryPolicySize
>> > > > > > >>>>>> to decrease memory allocated for each node.
>> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
>> > > settings:
>> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > HeapInit=250MB
>> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > HeapMax=3543MB
>> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > > > > >>>> DefaultMemoryPolicySize=
>> > > > > > >>>>>> 12753MB
>> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > > > > >>>>>> SystemCacheInitialSize=40MB
>> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > > > > >>>> SystemCacheMaxSize=100MB
>> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
>> ignite
>> > > > > > instanses
>> > > > > > >>>> on
>> > > > > > >>>>>> the server require: 12853MB
>> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
>> > > > internal.
>> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
>> > > skozlov@gridgain.com
>> > > > >:
>> > > > > > >>>>>>
>> > > > > > >>>>>>> I suppose we should not forget JVM heap size and
>> suggest to
>> > > > > reduce
>> > > > > > >>>> both
>> > > > > > >>>>>>> options
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>> > > > > > >>>>> dsetrakyan@apache.org
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>> wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> Igor, I would change the message. How about this:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Required RAM size is larger than total physical memory
>> > > > available
>> > > > > > for
>> > > > > > >>>>> OS.
>> > > > > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>> > > > > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
>> > memory
>> > > > > > >>>>> allocated
>> > > > > > >>>>>>> for
>> > > > > > >>>>>>>>> each node.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Also, can we calculate what the memory size allocated
>> for
>> > > each
>> > > > > > node
>> > > > > > >>>>>>> should
>> > > > > > >>>>>>>> be? In that case we should suggest it.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> D.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>> > > > > > >>>>> gvvinblade@gmail.com
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> wrote:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> What do you, guys think about next warning?
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO
>> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> > > GridHomePat
>> > > > > > >>>>>>> hSelfTest0]
>> > > > > > >>>>>>>>> System cache's MemoryPolicy size is configured to 40
>> MB.
>> > > Use
>> > > > > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
>> > > change
>> > > > > the
>> > > > > > >>>>>>>> setting.
>> > > > > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN
>> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> > > GridHomePat
>> > > > > > >>>>>>> hSelfTest0]
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>>>> Required RAM size is larger than total physical
>> memory
>> > > > > > >>>> available
>> > > > > > >>>>>>> for
>> > > > > > >>>>>>>>> OS.
>> > > > > > >>>>>>>>>>>> Check your configuration to avoid swap partition
>> > usage.
>> > > > > > >>>>>>>>>>>> Use MemoryConfiguration and
>> MemoryPolicyConfiguration
>> > to
>> > > > > > >>>> change
>> > > > > > >>>>>>> the
>> > > > > > >>>>>>>>> settings.
>> > > > > > >>>>>>>>>>>> Physical memory [required=16397MB,
>> available=15942MB]
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN
>> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> > > GridHomePat
>> > > > > > >>>>>>> hSelfTest0]
>> > > > > > >>>>>>>>> Peer class loading is enabled (disable it in
>> production
>> > for
>> > > > > > >>>>>>> performance
>> > > > > > >>>>>>>> and
>> > > > > > >>>>>>>>> deployment consistency reasons)
>> > > > > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO
>> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> > > GridHomePat
>> > > > > > >>>>>>> hSelfTest0]
>> > > > > > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>> > > > > > >>>>> ['ignite-sys-cache']]
>> > > > > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO
>> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
>> > > GridHomePat
>> > > > > > >>>>>>> hSelfTest0]
>> > > > > > >>>>>>>>> 3-rd party licenses can be found at:
>> > > > > > >>>>>>>>> /home/gvvinblade/projects/igni
>> te/incubator-ignite/libs/
>> > > > > licenses
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
>> > > > yzhdanov@apache.org
>> > > > > >:
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> Agree, let's release new version including tickets
>> > > mentioned
>> > > > > by
>> > > > > > >>>>>>> Denis
>> > > > > > >>>>>>>> and
>> > > > > > >>>>>>>>>> Ivan.
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> --Yakov
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> --
>> > > > > > >>>>>>> Sergey Kozlov
>> > > > > > >>>>>>> GridGain Systems
>> > > > > > >>>>>>> www.gridgain.com
>> > > > > > >>>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>
>> > > > > > >>>>
>> > > > > > >>
>> > > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Anton Vinogradov <av...@gridgain.com>.
Denis,
> BTW, who is considered to be the release manager of this release?
I'll do it.
On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor <gv...@gmail.com>
wrote:
> Ok, the check happens at the node start time or on NODE_JOIN event
>
> in general it looks like:
>
> 1) calculate expected used memory = heap max + system cache max + all
> custom policies max + default policy size and put it into a node attribute
>
> 2) get total physycal memory, calculate expected safe to be used memory
> amount (leave 4 gb min or 20% of available memory for OS)
>
> 3) if expected used memory + expected used memory of other nodes on the
> host > than safe to be used memory amount, start calculating suggestions
>
> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
> 100mb default polycy, if available memory is less we cannot suggest
> anything reasonable, print warning, stop calculation.
>
> 5) check heap size (shouldn't exceed 30% of available memory (total_memory
> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
> calculated value or 512MB minimal)
>
> 6) check if system cache size changed, suggest default value if it's so
>
> 7) in case 100 mb * policies count < available memory, suggest using
> default policy with max size equals to remaining memory (available - heap -
> system cache)
>
> 8) calculate new size for each memory policy ( it's user defined size *
> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> remaining memory, devided by nodes number on the host or 100 mb minimal)
>
> 9) print suggestions
>
>
>
> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
> > Igor, can you please describe the algorithm with all the thresholds?
> >
> > On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gv...@gmail.com>
> > wrote:
> >
> > > The suggestion here is based on initial settings, and it's so because
> > there
> > > is no other nodes on the host in the example.
> > >
> > > The algorithm tries to preserve the original ratio of memory policies
> > > keeping numbers reasonable (for example after some thresshold it will
> > > suggest not to use several memory policies if there is not enough of
> > memory
> > > for all of them) and taking into consideration nodes count on the host,
> > > each jvm heap, needed memory for OS, etc
> > >
> > > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> > >
> > > > Looks good, but why in the example provided are we suggesting 8GB? 2
> > > nodes
> > > > with 8GB will completely exhaust the available memory. I would
> suggest
> > 6
> > > or
> > > > 7GB.
> > > >
> > > > Also, why 100MB for default policy. Anything under 1GB seems too
> small.
> > > >
> > > > Can you please comment?
> > > >
> > > > D.
> > > >
> > > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> > gvvinblade@gmail.com>
> > > > wrote:
> > > >
> > > > > One more example of possible warning:
> > > > >
> > > > > -----------------------------------------------------
> > > > > Excessive memory usage by Ignite node process (performance may
> drop)
> > > > > [requested=44613MB, available=15942MB].
> > > > >
> > > > > Please tune the folowing settings as suggested:
> > > > > MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > > > > MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > > > >
> > > > > Current settings:
> > > > > Java Heap maxSize: 3543MB
> > > > > Java Heap initSize: 250MB
> > > > > MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > > > > MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > > > > The overall expected memory usage by all Ignite nodes on the
> host:
> > > > > 44613MB
> > > > > -----------------------------------------------------
> > > > >
> > > > > Your thoughts?
> > > > >
> > > > > 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > ML lib profile is missing in 2.1 release! That must be fixed and
> > > rolled
> > > > > > out in this emergency release:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > > > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > > > > >
> > > > > > Oleg, Yuri, please step in and handle the issue.
> > > > > >
> > > > > > BTW, who is considered to be the release manager of this release?
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > > >
> > > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > > > dsetrakyan@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > I like the format proposed by Denis, very clear.
> > > > > > >
> > > > > > > However, I also do not understand why a user should change the
> > size
> > > > of
> > > > > > some
> > > > > > > system cache. How would a user ever know what value to put
> there?
> > > > This
> > > > > > > value should be configured by Ignite automatically.
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <
> dmagda@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > >> Igor,
> > > > > > >>
> > > > > > >> Let me suggest this format.
> > > > > > >>
> > > > > > >> ---------------------------------------------
> > > > > > >> Excessive memory usage by Ignite node process (performance may
> > > drop)
> > > > > > >> [requested=29251MB, available=15942MB]
> > > > > > >>
> > > > > > >> Please tune the following settings:
> > > > > > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested
> > value]
> > > > > > >> MemoryConfiguration.systemCacheMaxSize = suggested value
> > > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> > > suggested
> > > > > > >> value]
> > > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> > > suggested
> > > > > > >> value]
> > > > > > >>
> > > > > > >> Current settings:
> > > > > > >> [DefaultMemoryPolicySize = value]
> > > > > > >> [{policy_name_1} size = value]
> > > > > > >> [{policy_name_1} size = value]
> > > > > > >> SystemCacheInitialSize = value
> > > > > > >> SystemCacheMaxSize = value
> > > > > > >> Java Heap Init Size = value
> > > > > > >> Java Heap Max Size = value
> > > > > > >>
> > > > > > >> The overall memory usage by all Ignite nodes on the host:
> value
> > > > > > >> -------------------------------------------
> > > > > > >>
> > > > > > >> Records in […] are optional. If custom memory policy is not
> set
> > or
> > > > the
> > > > > > >> default memory policy is overridden the output will miss some
> of
> > > the
> > > > > > rows.
> > > > > > >>
> > > > > > >> As for systemCacheMaxSize, it should be show ONLY if the
> > parameter
> > > > was
> > > > > > set
> > > > > > >> explicitly by user code. Otherwise, the platform should be
> wise
> > > > enough
> > > > > > to
> > > > > > >> instantiate it properly depending on the host memory usage.
> > > > > > >>
> > > > > > >> —
> > > > > > >> Denis
> > > > > > >>
> > > > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> > > > gvvinblade@gmail.com>
> > > > > > >> wrote:
> > > > > > >>>
> > > > > > >>> The message without logging layout:
> > > > > > >>>
> > > > > > >>> Not enough memory for current process [required=29251MB,
> > > > > > >> available=15942MB].
> > > > > > >>> Please change MemoryConfiguration.systemCacheMaxSize and
> > > > > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease
> memory
> > > > > > allocated
> > > > > > >>> for each node.
> > > > > > >>>
> > > > > > >>> Current settings:
> > > > > > >>> HeapInit=250MB
> > > > > > >>> HeapMax=3543MB
> > > > > > >>> DefaultMemoryPolicySize=12753MB
> > > > > > >>> SystemCacheInitialSize=40MB
> > > > > > >>> SystemCacheMaxSize=100MB
> > > > > > >>>
> > > > > > >>> Other ignite instanses on the server require: 12853MB
> > > > > > >>>
> > > > > > >>> I think it's make sense to describe what these numbers are
> > > consist
> > > > > of.
> > > > > > >>>
> > > > > > >>> We simple say which parameters have an impact on how much
> > memory
> > > > the
> > > > > > >>> instance needs and their (parameters) actual values.
> > > > > > >>>
> > > > > > >>> Also we notice that more than one Ignite instance are ran on
> > the
> > > > > server
> > > > > > >> or
> > > > > > >>> workstation and it also consumes memory.
> > > > > > >>>
> > > > > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > > > > > >>> dsetrakyan@apache.org> написал:
> > > > > > >>>
> > > > > > >>>> Igor, what is this flood of WARN messaging coming after the
> > > text?
> > > > > Are
> > > > > > we
> > > > > > >>>> really going to print this whole thing out?
> > > > > > >>>>
> > > > > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> > > > > > gvvinblade@gmail.com
> > > > > > >>>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>>> This message appears on topology change in case the
> available
> > > > > memory
> > > > > > is
> > > > > > >>>>> exceeded
> > > > > > >>>>>
> > > > > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> > > > gvvinblade@gmail.com
> > > > > >:
> > > > > > >>>>>
> > > > > > >>>>>> An example of current impl:
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
> > > memory
> > > > > for
> > > > > > >>>>>> current process [required=29251MB, available=15942MB].
> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please
> change
> > > > > > >>>>>> MemoryConfiguration.systemCacheMaxSize and
> > > MemoryConfiguration.
> > > > > > >>>>> defaultMemoryPolicySize
> > > > > > >>>>>> to decrease memory allocated for each node.
> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> > > settings:
> > > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > HeapInit=250MB
> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > HeapMax=3543MB
> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > > >>>> DefaultMemoryPolicySize=
> > > > > > >>>>>> 12753MB
> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > > >>>>>> SystemCacheInitialSize=40MB
> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > > >>>> SystemCacheMaxSize=100MB
> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other
> ignite
> > > > > > instanses
> > > > > > >>>> on
> > > > > > >>>>>> the server require: 12853MB
> > > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > > internal.
> > > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> > > skozlov@gridgain.com
> > > > >:
> > > > > > >>>>>>
> > > > > > >>>>>>> I suppose we should not forget JVM heap size and suggest
> to
> > > > > reduce
> > > > > > >>>> both
> > > > > > >>>>>>> options
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > > > > > >>>>> dsetrakyan@apache.org
> > > > > > >>>>>>>>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Igor, I would change the message. How about this:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Required RAM size is larger than total physical memory
> > > > available
> > > > > > for
> > > > > > >>>>> OS.
> > > > > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> > > > > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
> > memory
> > > > > > >>>>> allocated
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>> each node.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Also, can we calculate what the memory size allocated
> for
> > > each
> > > > > > node
> > > > > > >>>>>>> should
> > > > > > >>>>>>>> be? In that case we should suggest it.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> D.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > > > > > >>>>> gvvinblade@gmail.com
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> What do you, guys think about next warning?
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > GridHomePat
> > > > > > >>>>>>> hSelfTest0]
> > > > > > >>>>>>>>> System cache's MemoryPolicy size is configured to 40
> MB.
> > > Use
> > > > > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
> > > change
> > > > > the
> > > > > > >>>>>>>> setting.
> > > > > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > GridHomePat
> > > > > > >>>>>>> hSelfTest0]
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>>>> Required RAM size is larger than total physical
> memory
> > > > > > >>>> available
> > > > > > >>>>>>> for
> > > > > > >>>>>>>>> OS.
> > > > > > >>>>>>>>>>>> Check your configuration to avoid swap partition
> > usage.
> > > > > > >>>>>>>>>>>> Use MemoryConfiguration and
> MemoryPolicyConfiguration
> > to
> > > > > > >>>> change
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>> settings.
> > > > > > >>>>>>>>>>>> Physical memory [required=16397MB,
> available=15942MB]
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > GridHomePat
> > > > > > >>>>>>> hSelfTest0]
> > > > > > >>>>>>>>> Peer class loading is enabled (disable it in production
> > for
> > > > > > >>>>>>> performance
> > > > > > >>>>>>>> and
> > > > > > >>>>>>>>> deployment consistency reasons)
> > > > > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > GridHomePat
> > > > > > >>>>>>> hSelfTest0]
> > > > > > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> > > > > > >>>>> ['ignite-sys-cache']]
> > > > > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > > GridHomePat
> > > > > > >>>>>>> hSelfTest0]
> > > > > > >>>>>>>>> 3-rd party licenses can be found at:
> > > > > > >>>>>>>>> /home/gvvinblade/projects/
> ignite/incubator-ignite/libs/
> > > > > licenses
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> > > > yzhdanov@apache.org
> > > > > >:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Agree, let's release new version including tickets
> > > mentioned
> > > > > by
> > > > > > >>>>>>> Denis
> > > > > > >>>>>>>> and
> > > > > > >>>>>>>>>> Ivan.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> --Yakov
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> --
> > > > > > >>>>>>> Sergey Kozlov
> > > > > > >>>>>>> GridGain Systems
> > > > > > >>>>>>> www.gridgain.com
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Seliverstov Igor <gv...@gmail.com>.
Ok, the check happens at the node start time or on NODE_JOIN event
in general it looks like:
1) calculate expected used memory = heap max + system cache max + all
custom policies max + default policy size and put it into a node attribute
2) get total physycal memory, calculate expected safe to be used memory
amount (leave 4 gb min or 20% of available memory for OS)
3) if expected used memory + expected used memory of other nodes on the
host > than safe to be used memory amount, start calculating suggestions
4) Each ignite instance needs at least 512mb heap + 40mb system cache +
100mb default polycy, if available memory is less we cannot suggest
anything reasonable, print warning, stop calculation.
5) check heap size (shouldn't exceed 30% of available memory (total_memory
- reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
calculated value or 512MB minimal)
6) check if system cache size changed, suggest default value if it's so
7) in case 100 mb * policies count < available memory, suggest using
default policy with max size equals to remaining memory (available - heap -
system cache)
8) calculate new size for each memory policy ( it's user defined size *
(remaining / (all_policies_size * nodes_cnt)); in proportion to
remaining memory, devided by nodes number on the host or 100 mb minimal)
9) print suggestions
2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> Igor, can you please describe the algorithm with all the thresholds?
>
> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gv...@gmail.com>
> wrote:
>
> > The suggestion here is based on initial settings, and it's so because
> there
> > is no other nodes on the host in the example.
> >
> > The algorithm tries to preserve the original ratio of memory policies
> > keeping numbers reasonable (for example after some thresshold it will
> > suggest not to use several memory policies if there is not enough of
> memory
> > for all of them) and taking into consideration nodes count on the host,
> > each jvm heap, needed memory for OS, etc
> >
> > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> >
> > > Looks good, but why in the example provided are we suggesting 8GB? 2
> > nodes
> > > with 8GB will completely exhaust the available memory. I would suggest
> 6
> > or
> > > 7GB.
> > >
> > > Also, why 100MB for default policy. Anything under 1GB seems too small.
> > >
> > > Can you please comment?
> > >
> > > D.
> > >
> > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> gvvinblade@gmail.com>
> > > wrote:
> > >
> > > > One more example of possible warning:
> > > >
> > > > -----------------------------------------------------
> > > > Excessive memory usage by Ignite node process (performance may drop)
> > > > [requested=44613MB, available=15942MB].
> > > >
> > > > Please tune the folowing settings as suggested:
> > > > MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > > > MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > > >
> > > > Current settings:
> > > > Java Heap maxSize: 3543MB
> > > > Java Heap initSize: 250MB
> > > > MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > > > MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > > > The overall expected memory usage by all Ignite nodes on the host:
> > > > 44613MB
> > > > -----------------------------------------------------
> > > >
> > > > Your thoughts?
> > > >
> > > > 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> > > >
> > > > > Guys,
> > > > >
> > > > > ML lib profile is missing in 2.1 release! That must be fixed and
> > rolled
> > > > > out in this emergency release:
> > > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > > > >
> > > > > Oleg, Yuri, please step in and handle the issue.
> > > > >
> > > > > BTW, who is considered to be the release manager of this release?
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > I like the format proposed by Denis, very clear.
> > > > > >
> > > > > > However, I also do not understand why a user should change the
> size
> > > of
> > > > > some
> > > > > > system cache. How would a user ever know what value to put there?
> > > This
> > > > > > value should be configured by Ignite automatically.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dm...@apache.org>
> > > > wrote:
> > > > > >
> > > > > >> Igor,
> > > > > >>
> > > > > >> Let me suggest this format.
> > > > > >>
> > > > > >> ---------------------------------------------
> > > > > >> Excessive memory usage by Ignite node process (performance may
> > drop)
> > > > > >> [requested=29251MB, available=15942MB]
> > > > > >>
> > > > > >> Please tune the following settings:
> > > > > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested
> value]
> > > > > >> MemoryConfiguration.systemCacheMaxSize = suggested value
> > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> > suggested
> > > > > >> value]
> > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> > suggested
> > > > > >> value]
> > > > > >>
> > > > > >> Current settings:
> > > > > >> [DefaultMemoryPolicySize = value]
> > > > > >> [{policy_name_1} size = value]
> > > > > >> [{policy_name_1} size = value]
> > > > > >> SystemCacheInitialSize = value
> > > > > >> SystemCacheMaxSize = value
> > > > > >> Java Heap Init Size = value
> > > > > >> Java Heap Max Size = value
> > > > > >>
> > > > > >> The overall memory usage by all Ignite nodes on the host: value
> > > > > >> -------------------------------------------
> > > > > >>
> > > > > >> Records in […] are optional. If custom memory policy is not set
> or
> > > the
> > > > > >> default memory policy is overridden the output will miss some of
> > the
> > > > > rows.
> > > > > >>
> > > > > >> As for systemCacheMaxSize, it should be show ONLY if the
> parameter
> > > was
> > > > > set
> > > > > >> explicitly by user code. Otherwise, the platform should be wise
> > > enough
> > > > > to
> > > > > >> instantiate it properly depending on the host memory usage.
> > > > > >>
> > > > > >> —
> > > > > >> Denis
> > > > > >>
> > > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> > > gvvinblade@gmail.com>
> > > > > >> wrote:
> > > > > >>>
> > > > > >>> The message without logging layout:
> > > > > >>>
> > > > > >>> Not enough memory for current process [required=29251MB,
> > > > > >> available=15942MB].
> > > > > >>> Please change MemoryConfiguration.systemCacheMaxSize and
> > > > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> > > > > allocated
> > > > > >>> for each node.
> > > > > >>>
> > > > > >>> Current settings:
> > > > > >>> HeapInit=250MB
> > > > > >>> HeapMax=3543MB
> > > > > >>> DefaultMemoryPolicySize=12753MB
> > > > > >>> SystemCacheInitialSize=40MB
> > > > > >>> SystemCacheMaxSize=100MB
> > > > > >>>
> > > > > >>> Other ignite instanses on the server require: 12853MB
> > > > > >>>
> > > > > >>> I think it's make sense to describe what these numbers are
> > consist
> > > > of.
> > > > > >>>
> > > > > >>> We simple say which parameters have an impact on how much
> memory
> > > the
> > > > > >>> instance needs and their (parameters) actual values.
> > > > > >>>
> > > > > >>> Also we notice that more than one Ignite instance are ran on
> the
> > > > server
> > > > > >> or
> > > > > >>> workstation and it also consumes memory.
> > > > > >>>
> > > > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > > > > >>> dsetrakyan@apache.org> написал:
> > > > > >>>
> > > > > >>>> Igor, what is this flood of WARN messaging coming after the
> > text?
> > > > Are
> > > > > we
> > > > > >>>> really going to print this whole thing out?
> > > > > >>>>
> > > > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> > > > > gvvinblade@gmail.com
> > > > > >>>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> This message appears on topology change in case the available
> > > > memory
> > > > > is
> > > > > >>>>> exceeded
> > > > > >>>>>
> > > > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> > > gvvinblade@gmail.com
> > > > >:
> > > > > >>>>>
> > > > > >>>>>> An example of current impl:
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
> > memory
> > > > for
> > > > > >>>>>> current process [required=29251MB, available=15942MB].
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > > > > >>>>>> MemoryConfiguration.systemCacheMaxSize and
> > MemoryConfiguration.
> > > > > >>>>> defaultMemoryPolicySize
> > > > > >>>>>> to decrease memory allocated for each node.
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> > settings:
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > HeapInit=250MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > HeapMax=3543MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>> DefaultMemoryPolicySize=
> > > > > >>>>>> 12753MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>> SystemCacheInitialSize=40MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>> SystemCacheMaxSize=100MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite
> > > > > instanses
> > > > > >>>> on
> > > > > >>>>>> the server require: 12853MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> > skozlov@gridgain.com
> > > >:
> > > > > >>>>>>
> > > > > >>>>>>> I suppose we should not forget JVM heap size and suggest to
> > > > reduce
> > > > > >>>> both
> > > > > >>>>>>> options
> > > > > >>>>>>>
> > > > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > > > > >>>>> dsetrakyan@apache.org
> > > > > >>>>>>>>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Igor, I would change the message. How about this:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Required RAM size is larger than total physical memory
> > > available
> > > > > for
> > > > > >>>>> OS.
> > > > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> > > > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease
> memory
> > > > > >>>>> allocated
> > > > > >>>>>>> for
> > > > > >>>>>>>>> each node.
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> Also, can we calculate what the memory size allocated for
> > each
> > > > > node
> > > > > >>>>>>> should
> > > > > >>>>>>>> be? In that case we should suggest it.
> > > > > >>>>>>>>
> > > > > >>>>>>>> D.
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > > > > >>>>> gvvinblade@gmail.com
> > > > > >>>>>>>>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> What do you, guys think about next warning?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>> System cache's MemoryPolicy size is configured to 40 MB.
> > Use
> > > > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
> > change
> > > > the
> > > > > >>>>>>>> setting.
> > > > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>>>> Required RAM size is larger than total physical memory
> > > > > >>>> available
> > > > > >>>>>>> for
> > > > > >>>>>>>>> OS.
> > > > > >>>>>>>>>>>> Check your configuration to avoid swap partition
> usage.
> > > > > >>>>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration
> to
> > > > > >>>> change
> > > > > >>>>>>> the
> > > > > >>>>>>>>> settings.
> > > > > >>>>>>>>>>>> Physical memory [required=16397MB, available=15942MB]
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>> Peer class loading is enabled (disable it in production
> for
> > > > > >>>>>>> performance
> > > > > >>>>>>>> and
> > > > > >>>>>>>>> deployment consistency reasons)
> > > > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> > > > > >>>>> ['ignite-sys-cache']]
> > > > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>> 3-rd party licenses can be found at:
> > > > > >>>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/
> > > > licenses
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> > > yzhdanov@apache.org
> > > > >:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Agree, let's release new version including tickets
> > mentioned
> > > > by
> > > > > >>>>>>> Denis
> > > > > >>>>>>>> and
> > > > > >>>>>>>>>> Ivan.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> --Yakov
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> --
> > > > > >>>>>>> Sergey Kozlov
> > > > > >>>>>>> GridGain Systems
> > > > > >>>>>>> www.gridgain.com
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Igor, can you please describe the algorithm with all the thresholds?
On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gv...@gmail.com>
wrote:
> The suggestion here is based on initial settings, and it's so because there
> is no other nodes on the host in the example.
>
> The algorithm tries to preserve the original ratio of memory policies
> keeping numbers reasonable (for example after some thresshold it will
> suggest not to use several memory policies if there is not enough of memory
> for all of them) and taking into consideration nodes count on the host,
> each jvm heap, needed memory for OS, etc
>
> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
> > Looks good, but why in the example provided are we suggesting 8GB? 2
> nodes
> > with 8GB will completely exhaust the available memory. I would suggest 6
> or
> > 7GB.
> >
> > Also, why 100MB for default policy. Anything under 1GB seems too small.
> >
> > Can you please comment?
> >
> > D.
> >
> > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <gv...@gmail.com>
> > wrote:
> >
> > > One more example of possible warning:
> > >
> > > -----------------------------------------------------
> > > Excessive memory usage by Ignite node process (performance may drop)
> > > [requested=44613MB, available=15942MB].
> > >
> > > Please tune the folowing settings as suggested:
> > > MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > > MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > > MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > > MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > >
> > > Current settings:
> > > Java Heap maxSize: 3543MB
> > > Java Heap initSize: 250MB
> > > MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > > MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > > MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > > MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > > The overall expected memory usage by all Ignite nodes on the host:
> > > 44613MB
> > > -----------------------------------------------------
> > >
> > > Your thoughts?
> > >
> > > 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> > >
> > > > Guys,
> > > >
> > > > ML lib profile is missing in 2.1 release! That must be fixed and
> rolled
> > > > out in this emergency release:
> > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > > >
> > > > Oleg, Yuri, please step in and handle the issue.
> > > >
> > > > BTW, who is considered to be the release manager of this release?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > > wrote:
> > > > >
> > > > > I like the format proposed by Denis, very clear.
> > > > >
> > > > > However, I also do not understand why a user should change the size
> > of
> > > > some
> > > > > system cache. How would a user ever know what value to put there?
> > This
> > > > > value should be configured by Ignite automatically.
> > > > >
> > > > > D.
> > > > >
> > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dm...@apache.org>
> > > wrote:
> > > > >
> > > > >> Igor,
> > > > >>
> > > > >> Let me suggest this format.
> > > > >>
> > > > >> ---------------------------------------------
> > > > >> Excessive memory usage by Ignite node process (performance may
> drop)
> > > > >> [requested=29251MB, available=15942MB]
> > > > >>
> > > > >> Please tune the following settings:
> > > > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> > > > >> MemoryConfiguration.systemCacheMaxSize = suggested value
> > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> suggested
> > > > >> value]
> > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> suggested
> > > > >> value]
> > > > >>
> > > > >> Current settings:
> > > > >> [DefaultMemoryPolicySize = value]
> > > > >> [{policy_name_1} size = value]
> > > > >> [{policy_name_1} size = value]
> > > > >> SystemCacheInitialSize = value
> > > > >> SystemCacheMaxSize = value
> > > > >> Java Heap Init Size = value
> > > > >> Java Heap Max Size = value
> > > > >>
> > > > >> The overall memory usage by all Ignite nodes on the host: value
> > > > >> -------------------------------------------
> > > > >>
> > > > >> Records in […] are optional. If custom memory policy is not set or
> > the
> > > > >> default memory policy is overridden the output will miss some of
> the
> > > > rows.
> > > > >>
> > > > >> As for systemCacheMaxSize, it should be show ONLY if the parameter
> > was
> > > > set
> > > > >> explicitly by user code. Otherwise, the platform should be wise
> > enough
> > > > to
> > > > >> instantiate it properly depending on the host memory usage.
> > > > >>
> > > > >> —
> > > > >> Denis
> > > > >>
> > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> > gvvinblade@gmail.com>
> > > > >> wrote:
> > > > >>>
> > > > >>> The message without logging layout:
> > > > >>>
> > > > >>> Not enough memory for current process [required=29251MB,
> > > > >> available=15942MB].
> > > > >>> Please change MemoryConfiguration.systemCacheMaxSize and
> > > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> > > > allocated
> > > > >>> for each node.
> > > > >>>
> > > > >>> Current settings:
> > > > >>> HeapInit=250MB
> > > > >>> HeapMax=3543MB
> > > > >>> DefaultMemoryPolicySize=12753MB
> > > > >>> SystemCacheInitialSize=40MB
> > > > >>> SystemCacheMaxSize=100MB
> > > > >>>
> > > > >>> Other ignite instanses on the server require: 12853MB
> > > > >>>
> > > > >>> I think it's make sense to describe what these numbers are
> consist
> > > of.
> > > > >>>
> > > > >>> We simple say which parameters have an impact on how much memory
> > the
> > > > >>> instance needs and their (parameters) actual values.
> > > > >>>
> > > > >>> Also we notice that more than one Ignite instance are ran on the
> > > server
> > > > >> or
> > > > >>> workstation and it also consumes memory.
> > > > >>>
> > > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > > > >>> dsetrakyan@apache.org> написал:
> > > > >>>
> > > > >>>> Igor, what is this flood of WARN messaging coming after the
> text?
> > > Are
> > > > we
> > > > >>>> really going to print this whole thing out?
> > > > >>>>
> > > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> > > > gvvinblade@gmail.com
> > > > >>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> This message appears on topology change in case the available
> > > memory
> > > > is
> > > > >>>>> exceeded
> > > > >>>>>
> > > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> > gvvinblade@gmail.com
> > > >:
> > > > >>>>>
> > > > >>>>>> An example of current impl:
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough
> memory
> > > for
> > > > >>>>>> current process [required=29251MB, available=15942MB].
> > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > > > >>>>>> MemoryConfiguration.systemCacheMaxSize and
> MemoryConfiguration.
> > > > >>>>> defaultMemoryPolicySize
> > > > >>>>>> to decrease memory allocated for each node.
> > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current
> settings:
> > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> HeapInit=250MB
> > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> HeapMax=3543MB
> > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > >>>> DefaultMemoryPolicySize=
> > > > >>>>>> 12753MB
> > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > >>>>>> SystemCacheInitialSize=40MB
> > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > >>>> SystemCacheMaxSize=100MB
> > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite
> > > > instanses
> > > > >>>> on
> > > > >>>>>> the server require: 12853MB
> > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > internal.
> > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <
> skozlov@gridgain.com
> > >:
> > > > >>>>>>
> > > > >>>>>>> I suppose we should not forget JVM heap size and suggest to
> > > reduce
> > > > >>>> both
> > > > >>>>>>> options
> > > > >>>>>>>
> > > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > > > >>>>> dsetrakyan@apache.org
> > > > >>>>>>>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Igor, I would change the message. How about this:
> > > > >>>>>>>>
> > > > >>>>>>>> Required RAM size is larger than total physical memory
> > available
> > > > for
> > > > >>>>> OS.
> > > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> > > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease memory
> > > > >>>>> allocated
> > > > >>>>>>> for
> > > > >>>>>>>>> each node.
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> Also, can we calculate what the memory size allocated for
> each
> > > > node
> > > > >>>>>>> should
> > > > >>>>>>>> be? In that case we should suggest it.
> > > > >>>>>>>>
> > > > >>>>>>>> D.
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > > > >>>>> gvvinblade@gmail.com
> > > > >>>>>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> What do you, guys think about next warning?
> > > > >>>>>>>>>
> > > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePat
> > > > >>>>>>> hSelfTest0]
> > > > >>>>>>>>> System cache's MemoryPolicy size is configured to 40 MB.
> Use
> > > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to
> change
> > > the
> > > > >>>>>>>> setting.
> > > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePat
> > > > >>>>>>> hSelfTest0]
> > > > >>>>>>>>>
> > > > >>>>>>>>>>>> Required RAM size is larger than total physical memory
> > > > >>>> available
> > > > >>>>>>> for
> > > > >>>>>>>>> OS.
> > > > >>>>>>>>>>>> Check your configuration to avoid swap partition usage.
> > > > >>>>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration to
> > > > >>>> change
> > > > >>>>>>> the
> > > > >>>>>>>>> settings.
> > > > >>>>>>>>>>>> Physical memory [required=16397MB, available=15942MB]
> > > > >>>>>>>>>
> > > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePat
> > > > >>>>>>> hSelfTest0]
> > > > >>>>>>>>> Peer class loading is enabled (disable it in production for
> > > > >>>>>>> performance
> > > > >>>>>>>> and
> > > > >>>>>>>>> deployment consistency reasons)
> > > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePat
> > > > >>>>>>> hSelfTest0]
> > > > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> > > > >>>>> ['ignite-sys-cache']]
> > > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePat
> > > > >>>>>>> hSelfTest0]
> > > > >>>>>>>>> 3-rd party licenses can be found at:
> > > > >>>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/
> > > licenses
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> > yzhdanov@apache.org
> > > >:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Agree, let's release new version including tickets
> mentioned
> > > by
> > > > >>>>>>> Denis
> > > > >>>>>>>> and
> > > > >>>>>>>>>> Ivan.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> --Yakov
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> --
> > > > >>>>>>> Sergey Kozlov
> > > > >>>>>>> GridGain Systems
> > > > >>>>>>> www.gridgain.com
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Seliverstov Igor <gv...@gmail.com>.
The suggestion here is based on initial settings, and it's so because there
is no other nodes on the host in the example.
The algorithm tries to preserve the original ratio of memory policies
keeping numbers reasonable (for example after some thresshold it will
suggest not to use several memory policies if there is not enough of memory
for all of them) and taking into consideration nodes count on the host,
each jvm heap, needed memory for OS, etc
2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> Looks good, but why in the example provided are we suggesting 8GB? 2 nodes
> with 8GB will completely exhaust the available memory. I would suggest 6 or
> 7GB.
>
> Also, why 100MB for default policy. Anything under 1GB seems too small.
>
> Can you please comment?
>
> D.
>
> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <gv...@gmail.com>
> wrote:
>
> > One more example of possible warning:
> >
> > -----------------------------------------------------
> > Excessive memory usage by Ignite node process (performance may drop)
> > [requested=44613MB, available=15942MB].
> >
> > Please tune the folowing settings as suggested:
> > MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> >
> > Current settings:
> > Java Heap maxSize: 3543MB
> > Java Heap initSize: 250MB
> > MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > The overall expected memory usage by all Ignite nodes on the host:
> > 44613MB
> > -----------------------------------------------------
> >
> > Your thoughts?
> >
> > 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> >
> > > Guys,
> > >
> > > ML lib profile is missing in 2.1 release! That must be fixed and rolled
> > > out in this emergency release:
> > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > >
> > > Oleg, Yuri, please step in and handle the issue.
> > >
> > > BTW, who is considered to be the release manager of this release?
> > >
> > > —
> > > Denis
> > >
> > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > > wrote:
> > > >
> > > > I like the format proposed by Denis, very clear.
> > > >
> > > > However, I also do not understand why a user should change the size
> of
> > > some
> > > > system cache. How would a user ever know what value to put there?
> This
> > > > value should be configured by Ignite automatically.
> > > >
> > > > D.
> > > >
> > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dm...@apache.org>
> > wrote:
> > > >
> > > >> Igor,
> > > >>
> > > >> Let me suggest this format.
> > > >>
> > > >> ---------------------------------------------
> > > >> Excessive memory usage by Ignite node process (performance may drop)
> > > >> [requested=29251MB, available=15942MB]
> > > >>
> > > >> Please tune the following settings:
> > > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> > > >> MemoryConfiguration.systemCacheMaxSize = suggested value
> > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> > > >> value]
> > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> > > >> value]
> > > >>
> > > >> Current settings:
> > > >> [DefaultMemoryPolicySize = value]
> > > >> [{policy_name_1} size = value]
> > > >> [{policy_name_1} size = value]
> > > >> SystemCacheInitialSize = value
> > > >> SystemCacheMaxSize = value
> > > >> Java Heap Init Size = value
> > > >> Java Heap Max Size = value
> > > >>
> > > >> The overall memory usage by all Ignite nodes on the host: value
> > > >> -------------------------------------------
> > > >>
> > > >> Records in […] are optional. If custom memory policy is not set or
> the
> > > >> default memory policy is overridden the output will miss some of the
> > > rows.
> > > >>
> > > >> As for systemCacheMaxSize, it should be show ONLY if the parameter
> was
> > > set
> > > >> explicitly by user code. Otherwise, the platform should be wise
> enough
> > > to
> > > >> instantiate it properly depending on the host memory usage.
> > > >>
> > > >> —
> > > >> Denis
> > > >>
> > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> gvvinblade@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>> The message without logging layout:
> > > >>>
> > > >>> Not enough memory for current process [required=29251MB,
> > > >> available=15942MB].
> > > >>> Please change MemoryConfiguration.systemCacheMaxSize and
> > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> > > allocated
> > > >>> for each node.
> > > >>>
> > > >>> Current settings:
> > > >>> HeapInit=250MB
> > > >>> HeapMax=3543MB
> > > >>> DefaultMemoryPolicySize=12753MB
> > > >>> SystemCacheInitialSize=40MB
> > > >>> SystemCacheMaxSize=100MB
> > > >>>
> > > >>> Other ignite instanses on the server require: 12853MB
> > > >>>
> > > >>> I think it's make sense to describe what these numbers are consist
> > of.
> > > >>>
> > > >>> We simple say which parameters have an impact on how much memory
> the
> > > >>> instance needs and their (parameters) actual values.
> > > >>>
> > > >>> Also we notice that more than one Ignite instance are ran on the
> > server
> > > >> or
> > > >>> workstation and it also consumes memory.
> > > >>>
> > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > > >>> dsetrakyan@apache.org> написал:
> > > >>>
> > > >>>> Igor, what is this flood of WARN messaging coming after the text?
> > Are
> > > we
> > > >>>> really going to print this whole thing out?
> > > >>>>
> > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> > > gvvinblade@gmail.com
> > > >>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> This message appears on topology change in case the available
> > memory
> > > is
> > > >>>>> exceeded
> > > >>>>>
> > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <
> gvvinblade@gmail.com
> > >:
> > > >>>>>
> > > >>>>>> An example of current impl:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory
> > for
> > > >>>>>> current process [required=29251MB, available=15942MB].
> > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > > >>>>>> MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> > > >>>>> defaultMemoryPolicySize
> > > >>>>>> to decrease memory allocated for each node.
> > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
> > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
> > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >>>> DefaultMemoryPolicySize=
> > > >>>>>> 12753MB
> > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >>>>>> SystemCacheInitialSize=40MB
> > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >>>> SystemCacheMaxSize=100MB
> > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite
> > > instanses
> > > >>>> on
> > > >>>>>> the server require: 12853MB
> > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> internal.
> > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <skozlov@gridgain.com
> >:
> > > >>>>>>
> > > >>>>>>> I suppose we should not forget JVM heap size and suggest to
> > reduce
> > > >>>> both
> > > >>>>>>> options
> > > >>>>>>>
> > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > > >>>>> dsetrakyan@apache.org
> > > >>>>>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Igor, I would change the message. How about this:
> > > >>>>>>>>
> > > >>>>>>>> Required RAM size is larger than total physical memory
> available
> > > for
> > > >>>>> OS.
> > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease memory
> > > >>>>> allocated
> > > >>>>>>> for
> > > >>>>>>>>> each node.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Also, can we calculate what the memory size allocated for each
> > > node
> > > >>>>>>> should
> > > >>>>>>>> be? In that case we should suggest it.
> > > >>>>>>>>
> > > >>>>>>>> D.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > > >>>>> gvvinblade@gmail.com
> > > >>>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> What do you, guys think about next warning?
> > > >>>>>>>>>
> > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > > >>>>>>> hSelfTest0]
> > > >>>>>>>>> System cache's MemoryPolicy size is configured to 40 MB. Use
> > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to change
> > the
> > > >>>>>>>> setting.
> > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > > >>>>>>> hSelfTest0]
> > > >>>>>>>>>
> > > >>>>>>>>>>>> Required RAM size is larger than total physical memory
> > > >>>> available
> > > >>>>>>> for
> > > >>>>>>>>> OS.
> > > >>>>>>>>>>>> Check your configuration to avoid swap partition usage.
> > > >>>>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration to
> > > >>>> change
> > > >>>>>>> the
> > > >>>>>>>>> settings.
> > > >>>>>>>>>>>> Physical memory [required=16397MB, available=15942MB]
> > > >>>>>>>>>
> > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > > >>>>>>> hSelfTest0]
> > > >>>>>>>>> Peer class loading is enabled (disable it in production for
> > > >>>>>>> performance
> > > >>>>>>>> and
> > > >>>>>>>>> deployment consistency reasons)
> > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > > >>>>>>> hSelfTest0]
> > > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> > > >>>>> ['ignite-sys-cache']]
> > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > > >>>>>>> hSelfTest0]
> > > >>>>>>>>> 3-rd party licenses can be found at:
> > > >>>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/
> > licenses
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <
> yzhdanov@apache.org
> > >:
> > > >>>>>>>>>
> > > >>>>>>>>>> Agree, let's release new version including tickets mentioned
> > by
> > > >>>>>>> Denis
> > > >>>>>>>> and
> > > >>>>>>>>>> Ivan.
> > > >>>>>>>>>>
> > > >>>>>>>>>> --Yakov
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Sergey Kozlov
> > > >>>>>>> GridGain Systems
> > > >>>>>>> www.gridgain.com
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Looks good, but why in the example provided are we suggesting 8GB? 2 nodes
with 8GB will completely exhaust the available memory. I would suggest 6 or
7GB.
Also, why 100MB for default policy. Anything under 1GB seems too small.
Can you please comment?
D.
On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <gv...@gmail.com>
wrote:
> One more example of possible warning:
>
> -----------------------------------------------------
> Excessive memory usage by Ignite node process (performance may drop)
> [requested=44613MB, available=15942MB].
>
> Please tune the folowing settings as suggested:
> MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>
> Current settings:
> Java Heap maxSize: 3543MB
> Java Heap initSize: 250MB
> MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> The overall expected memory usage by all Ignite nodes on the host:
> 44613MB
> -----------------------------------------------------
>
> Your thoughts?
>
> 2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
>
> > Guys,
> >
> > ML lib profile is missing in 2.1 release! That must be fixed and rolled
> > out in this emergency release:
> > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > https://issues.apache.org/jira/browse/IGNITE-6193>
> >
> > Oleg, Yuri, please step in and handle the issue.
> >
> > BTW, who is considered to be the release manager of this release?
> >
> > —
> > Denis
> >
> > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <ds...@apache.org>
> > wrote:
> > >
> > > I like the format proposed by Denis, very clear.
> > >
> > > However, I also do not understand why a user should change the size of
> > some
> > > system cache. How would a user ever know what value to put there? This
> > > value should be configured by Ignite automatically.
> > >
> > > D.
> > >
> > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dm...@apache.org>
> wrote:
> > >
> > >> Igor,
> > >>
> > >> Let me suggest this format.
> > >>
> > >> ---------------------------------------------
> > >> Excessive memory usage by Ignite node process (performance may drop)
> > >> [requested=29251MB, available=15942MB]
> > >>
> > >> Please tune the following settings:
> > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> > >> MemoryConfiguration.systemCacheMaxSize = suggested value
> > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> > >> value]
> > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> > >> value]
> > >>
> > >> Current settings:
> > >> [DefaultMemoryPolicySize = value]
> > >> [{policy_name_1} size = value]
> > >> [{policy_name_1} size = value]
> > >> SystemCacheInitialSize = value
> > >> SystemCacheMaxSize = value
> > >> Java Heap Init Size = value
> > >> Java Heap Max Size = value
> > >>
> > >> The overall memory usage by all Ignite nodes on the host: value
> > >> -------------------------------------------
> > >>
> > >> Records in […] are optional. If custom memory policy is not set or the
> > >> default memory policy is overridden the output will miss some of the
> > rows.
> > >>
> > >> As for systemCacheMaxSize, it should be show ONLY if the parameter was
> > set
> > >> explicitly by user code. Otherwise, the platform should be wise enough
> > to
> > >> instantiate it properly depending on the host memory usage.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <gv...@gmail.com>
> > >> wrote:
> > >>>
> > >>> The message without logging layout:
> > >>>
> > >>> Not enough memory for current process [required=29251MB,
> > >> available=15942MB].
> > >>> Please change MemoryConfiguration.systemCacheMaxSize and
> > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> > allocated
> > >>> for each node.
> > >>>
> > >>> Current settings:
> > >>> HeapInit=250MB
> > >>> HeapMax=3543MB
> > >>> DefaultMemoryPolicySize=12753MB
> > >>> SystemCacheInitialSize=40MB
> > >>> SystemCacheMaxSize=100MB
> > >>>
> > >>> Other ignite instanses on the server require: 12853MB
> > >>>
> > >>> I think it's make sense to describe what these numbers are consist
> of.
> > >>>
> > >>> We simple say which parameters have an impact on how much memory the
> > >>> instance needs and their (parameters) actual values.
> > >>>
> > >>> Also we notice that more than one Ignite instance are ran on the
> server
> > >> or
> > >>> workstation and it also consumes memory.
> > >>>
> > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > >>> dsetrakyan@apache.org> написал:
> > >>>
> > >>>> Igor, what is this flood of WARN messaging coming after the text?
> Are
> > we
> > >>>> really going to print this whole thing out?
> > >>>>
> > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> > gvvinblade@gmail.com
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> This message appears on topology change in case the available
> memory
> > is
> > >>>>> exceeded
> > >>>>>
> > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <gvvinblade@gmail.com
> >:
> > >>>>>
> > >>>>>> An example of current impl:
> > >>>>>>
> > >>>>>>
> > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory
> for
> > >>>>>> current process [required=29251MB, available=15942MB].
> > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > >>>>>> MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> > >>>>> defaultMemoryPolicySize
> > >>>>>> to decrease memory allocated for each node.
> > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
> > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
> > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>> DefaultMemoryPolicySize=
> > >>>>>> 12753MB
> > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>> SystemCacheInitialSize=40MB
> > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>> SystemCacheMaxSize=100MB
> > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite
> > instanses
> > >>>> on
> > >>>>>> the server require: 12853MB
> > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > >>>>>>
> > >>>>>>
> > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
> > >>>>>>
> > >>>>>>> I suppose we should not forget JVM heap size and suggest to
> reduce
> > >>>> both
> > >>>>>>> options
> > >>>>>>>
> > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > >>>>> dsetrakyan@apache.org
> > >>>>>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Igor, I would change the message. How about this:
> > >>>>>>>>
> > >>>>>>>> Required RAM size is larger than total physical memory available
> > for
> > >>>>> OS.
> > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease memory
> > >>>>> allocated
> > >>>>>>> for
> > >>>>>>>>> each node.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Also, can we calculate what the memory size allocated for each
> > node
> > >>>>>>> should
> > >>>>>>>> be? In that case we should suggest it.
> > >>>>>>>>
> > >>>>>>>> D.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > >>>>> gvvinblade@gmail.com
> > >>>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> What do you, guys think about next warning?
> > >>>>>>>>>
> > >>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >>>>>>> hSelfTest0]
> > >>>>>>>>> System cache's MemoryPolicy size is configured to 40 MB. Use
> > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to change
> the
> > >>>>>>>> setting.
> > >>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >>>>>>> hSelfTest0]
> > >>>>>>>>>
> > >>>>>>>>>>>> Required RAM size is larger than total physical memory
> > >>>> available
> > >>>>>>> for
> > >>>>>>>>> OS.
> > >>>>>>>>>>>> Check your configuration to avoid swap partition usage.
> > >>>>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration to
> > >>>> change
> > >>>>>>> the
> > >>>>>>>>> settings.
> > >>>>>>>>>>>> Physical memory [required=16397MB, available=15942MB]
> > >>>>>>>>>
> > >>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >>>>>>> hSelfTest0]
> > >>>>>>>>> Peer class loading is enabled (disable it in production for
> > >>>>>>> performance
> > >>>>>>>> and
> > >>>>>>>>> deployment consistency reasons)
> > >>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >>>>>>> hSelfTest0]
> > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> > >>>>> ['ignite-sys-cache']]
> > >>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >>>>>>> hSelfTest0]
> > >>>>>>>>> 3-rd party licenses can be found at:
> > >>>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/
> licenses
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yzhdanov@apache.org
> >:
> > >>>>>>>>>
> > >>>>>>>>>> Agree, let's release new version including tickets mentioned
> by
> > >>>>>>> Denis
> > >>>>>>>> and
> > >>>>>>>>>> Ivan.
> > >>>>>>>>>>
> > >>>>>>>>>> --Yakov
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> Sergey Kozlov
> > >>>>>>> GridGain Systems
> > >>>>>>> www.gridgain.com
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Seliverstov Igor <gv...@gmail.com>.
One more example of possible warning:
-----------------------------------------------------
Excessive memory usage by Ignite node process (performance may drop)
[requested=44613MB, available=15942MB].
Please tune the folowing settings as suggested:
MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
Current settings:
Java Heap maxSize: 3543MB
Java Heap initSize: 250MB
MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
The overall expected memory usage by all Ignite nodes on the host: 44613MB
-----------------------------------------------------
Your thoughts?
2017-08-28 5:06 GMT+03:00 Denis Magda <dm...@apache.org>:
> Guys,
>
> ML lib profile is missing in 2.1 release! That must be fixed and rolled
> out in this emergency release:
> https://issues.apache.org/jira/browse/IGNITE-6193 <
> https://issues.apache.org/jira/browse/IGNITE-6193>
>
> Oleg, Yuri, please step in and handle the issue.
>
> BTW, who is considered to be the release manager of this release?
>
> —
> Denis
>
> > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> >
> > I like the format proposed by Denis, very clear.
> >
> > However, I also do not understand why a user should change the size of
> some
> > system cache. How would a user ever know what value to put there? This
> > value should be configured by Ignite automatically.
> >
> > D.
> >
> > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dm...@apache.org> wrote:
> >
> >> Igor,
> >>
> >> Let me suggest this format.
> >>
> >> ---------------------------------------------
> >> Excessive memory usage by Ignite node process (performance may drop)
> >> [requested=29251MB, available=15942MB]
> >>
> >> Please tune the following settings:
> >> [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> >> MemoryConfiguration.systemCacheMaxSize = suggested value
> >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> >> value]
> >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> >> value]
> >>
> >> Current settings:
> >> [DefaultMemoryPolicySize = value]
> >> [{policy_name_1} size = value]
> >> [{policy_name_1} size = value]
> >> SystemCacheInitialSize = value
> >> SystemCacheMaxSize = value
> >> Java Heap Init Size = value
> >> Java Heap Max Size = value
> >>
> >> The overall memory usage by all Ignite nodes on the host: value
> >> -------------------------------------------
> >>
> >> Records in […] are optional. If custom memory policy is not set or the
> >> default memory policy is overridden the output will miss some of the
> rows.
> >>
> >> As for systemCacheMaxSize, it should be show ONLY if the parameter was
> set
> >> explicitly by user code. Otherwise, the platform should be wise enough
> to
> >> instantiate it properly depending on the host memory usage.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <gv...@gmail.com>
> >> wrote:
> >>>
> >>> The message without logging layout:
> >>>
> >>> Not enough memory for current process [required=29251MB,
> >> available=15942MB].
> >>> Please change MemoryConfiguration.systemCacheMaxSize and
> >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> allocated
> >>> for each node.
> >>>
> >>> Current settings:
> >>> HeapInit=250MB
> >>> HeapMax=3543MB
> >>> DefaultMemoryPolicySize=12753MB
> >>> SystemCacheInitialSize=40MB
> >>> SystemCacheMaxSize=100MB
> >>>
> >>> Other ignite instanses on the server require: 12853MB
> >>>
> >>> I think it's make sense to describe what these numbers are consist of.
> >>>
> >>> We simple say which parameters have an impact on how much memory the
> >>> instance needs and their (parameters) actual values.
> >>>
> >>> Also we notice that more than one Ignite instance are ran on the server
> >> or
> >>> workstation and it also consumes memory.
> >>>
> >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> >>> dsetrakyan@apache.org> написал:
> >>>
> >>>> Igor, what is this flood of WARN messaging coming after the text? Are
> we
> >>>> really going to print this whole thing out?
> >>>>
> >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> gvvinblade@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> This message appears on topology change in case the available memory
> is
> >>>>> exceeded
> >>>>>
> >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <gv...@gmail.com>:
> >>>>>
> >>>>>> An example of current impl:
> >>>>>>
> >>>>>>
> >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> >>>>>> current process [required=29251MB, available=15942MB].
> >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> >>>>>> MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> >>>>> defaultMemoryPolicySize
> >>>>>> to decrease memory allocated for each node.
> >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
> >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
> >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>> DefaultMemoryPolicySize=
> >>>>>> 12753MB
> >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>> SystemCacheInitialSize=40MB
> >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>> SystemCacheMaxSize=100MB
> >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite
> instanses
> >>>> on
> >>>>>> the server require: 12853MB
> >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>>>
> >>>>>>
> >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
> >>>>>>
> >>>>>>> I suppose we should not forget JVM heap size and suggest to reduce
> >>>> both
> >>>>>>> options
> >>>>>>>
> >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> >>>>> dsetrakyan@apache.org
> >>>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Igor, I would change the message. How about this:
> >>>>>>>>
> >>>>>>>> Required RAM size is larger than total physical memory available
> for
> >>>>> OS.
> >>>>>>>>> Please change MemoryConfiguration.WhichProperty and
> >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease memory
> >>>>> allocated
> >>>>>>> for
> >>>>>>>>> each node.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Also, can we calculate what the memory size allocated for each
> node
> >>>>>>> should
> >>>>>>>> be? In that case we should suggest it.
> >>>>>>>>
> >>>>>>>> D.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> >>>>> gvvinblade@gmail.com
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> What do you, guys think about next warning?
> >>>>>>>>>
> >>>>>>>>> [2017-08-25 17:17:04,718][INFO
> >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>>>> hSelfTest0]
> >>>>>>>>> System cache's MemoryPolicy size is configured to 40 MB. Use
> >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to change the
> >>>>>>>> setting.
> >>>>>>>>> [2017-08-25 17:17:04,718][WARN
> >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>>>> hSelfTest0]
> >>>>>>>>>
> >>>>>>>>>>>> Required RAM size is larger than total physical memory
> >>>> available
> >>>>>>> for
> >>>>>>>>> OS.
> >>>>>>>>>>>> Check your configuration to avoid swap partition usage.
> >>>>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration to
> >>>> change
> >>>>>>> the
> >>>>>>>>> settings.
> >>>>>>>>>>>> Physical memory [required=16397MB, available=15942MB]
> >>>>>>>>>
> >>>>>>>>> [2017-08-25 17:17:04,726][WARN
> >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>>>> hSelfTest0]
> >>>>>>>>> Peer class loading is enabled (disable it in production for
> >>>>>>> performance
> >>>>>>>> and
> >>>>>>>>> deployment consistency reasons)
> >>>>>>>>> [2017-08-25 17:17:04,726][INFO
> >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>>>> hSelfTest0]
> >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> >>>>> ['ignite-sys-cache']]
> >>>>>>>>> [2017-08-25 17:17:04,731][INFO
> >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>>>> hSelfTest0]
> >>>>>>>>> 3-rd party licenses can be found at:
> >>>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
> >>>>>>>>>
> >>>>>>>>>> Agree, let's release new version including tickets mentioned by
> >>>>>>> Denis
> >>>>>>>> and
> >>>>>>>>>> Ivan.
> >>>>>>>>>>
> >>>>>>>>>> --Yakov
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Sergey Kozlov
> >>>>>>> GridGain Systems
> >>>>>>> www.gridgain.com
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Denis Magda <dm...@apache.org>.
Guys,
ML lib profile is missing in 2.1 release! That must be fixed and rolled out in this emergency release:
https://issues.apache.org/jira/browse/IGNITE-6193 <https://issues.apache.org/jira/browse/IGNITE-6193>
Oleg, Yuri, please step in and handle the issue.
BTW, who is considered to be the release manager of this release?
—
Denis
> On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> I like the format proposed by Denis, very clear.
>
> However, I also do not understand why a user should change the size of some
> system cache. How would a user ever know what value to put there? This
> value should be configured by Ignite automatically.
>
> D.
>
> On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Igor,
>>
>> Let me suggest this format.
>>
>> ---------------------------------------------
>> Excessive memory usage by Ignite node process (performance may drop)
>> [requested=29251MB, available=15942MB]
>>
>> Please tune the following settings:
>> [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
>> MemoryConfiguration.systemCacheMaxSize = suggested value
>> [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
>> value]
>> [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
>> value]
>>
>> Current settings:
>> [DefaultMemoryPolicySize = value]
>> [{policy_name_1} size = value]
>> [{policy_name_1} size = value]
>> SystemCacheInitialSize = value
>> SystemCacheMaxSize = value
>> Java Heap Init Size = value
>> Java Heap Max Size = value
>>
>> The overall memory usage by all Ignite nodes on the host: value
>> -------------------------------------------
>>
>> Records in […] are optional. If custom memory policy is not set or the
>> default memory policy is overridden the output will miss some of the rows.
>>
>> As for systemCacheMaxSize, it should be show ONLY if the parameter was set
>> explicitly by user code. Otherwise, the platform should be wise enough to
>> instantiate it properly depending on the host memory usage.
>>
>> —
>> Denis
>>
>>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <gv...@gmail.com>
>> wrote:
>>>
>>> The message without logging layout:
>>>
>>> Not enough memory for current process [required=29251MB,
>> available=15942MB].
>>> Please change MemoryConfiguration.systemCacheMaxSize and
>>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
>>> for each node.
>>>
>>> Current settings:
>>> HeapInit=250MB
>>> HeapMax=3543MB
>>> DefaultMemoryPolicySize=12753MB
>>> SystemCacheInitialSize=40MB
>>> SystemCacheMaxSize=100MB
>>>
>>> Other ignite instanses on the server require: 12853MB
>>>
>>> I think it's make sense to describe what these numbers are consist of.
>>>
>>> We simple say which parameters have an impact on how much memory the
>>> instance needs and their (parameters) actual values.
>>>
>>> Also we notice that more than one Ignite instance are ran on the server
>> or
>>> workstation and it also consumes memory.
>>>
>>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
>>> dsetrakyan@apache.org> написал:
>>>
>>>> Igor, what is this flood of WARN messaging coming after the text? Are we
>>>> really going to print this whole thing out?
>>>>
>>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <gvvinblade@gmail.com
>>>
>>>> wrote:
>>>>
>>>>> This message appears on topology change in case the available memory is
>>>>> exceeded
>>>>>
>>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <gv...@gmail.com>:
>>>>>
>>>>>> An example of current impl:
>>>>>>
>>>>>>
>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
>>>>>> current process [required=29251MB, available=15942MB].
>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
>>>>>> MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
>>>>> defaultMemoryPolicySize
>>>>>> to decrease memory allocated for each node.
>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
>>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>> DefaultMemoryPolicySize=
>>>>>> 12753MB
>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>> SystemCacheInitialSize=40MB
>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>> SystemCacheMaxSize=100MB
>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses
>>>> on
>>>>>> the server require: 12853MB
>>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>>>
>>>>>>
>>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
>>>>>>
>>>>>>> I suppose we should not forget JVM heap size and suggest to reduce
>>>> both
>>>>>>> options
>>>>>>>
>>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org
>>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Igor, I would change the message. How about this:
>>>>>>>>
>>>>>>>> Required RAM size is larger than total physical memory available for
>>>>> OS.
>>>>>>>>> Please change MemoryConfiguration.WhichProperty and
>>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease memory
>>>>> allocated
>>>>>>> for
>>>>>>>>> each node.
>>>>>>>>
>>>>>>>>
>>>>>>>> Also, can we calculate what the memory size allocated for each node
>>>>>>> should
>>>>>>>> be? In that case we should suggest it.
>>>>>>>>
>>>>>>>> D.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>>>> gvvinblade@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> What do you, guys think about next warning?
>>>>>>>>>
>>>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>>>> hSelfTest0]
>>>>>>>>> System cache's MemoryPolicy size is configured to 40 MB. Use
>>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to change the
>>>>>>>> setting.
>>>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>>>> hSelfTest0]
>>>>>>>>>
>>>>>>>>>>>> Required RAM size is larger than total physical memory
>>>> available
>>>>>>> for
>>>>>>>>> OS.
>>>>>>>>>>>> Check your configuration to avoid swap partition usage.
>>>>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration to
>>>> change
>>>>>>> the
>>>>>>>>> settings.
>>>>>>>>>>>> Physical memory [required=16397MB, available=15942MB]
>>>>>>>>>
>>>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>>>> hSelfTest0]
>>>>>>>>> Peer class loading is enabled (disable it in production for
>>>>>>> performance
>>>>>>>> and
>>>>>>>>> deployment consistency reasons)
>>>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>>>> hSelfTest0]
>>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>>>> ['ignite-sys-cache']]
>>>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>>>> hSelfTest0]
>>>>>>>>> 3-rd party licenses can be found at:
>>>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
>>>>>>>>>
>>>>>>>>>> Agree, let's release new version including tickets mentioned by
>>>>>>> Denis
>>>>>>>> and
>>>>>>>>>> Ivan.
>>>>>>>>>>
>>>>>>>>>> --Yakov
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sergey Kozlov
>>>>>>> GridGain Systems
>>>>>>> www.gridgain.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
I like the format proposed by Denis, very clear.
However, I also do not understand why a user should change the size of some
system cache. How would a user ever know what value to put there? This
value should be configured by Ignite automatically.
D.
On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dm...@apache.org> wrote:
> Igor,
>
> Let me suggest this format.
>
> ---------------------------------------------
> Excessive memory usage by Ignite node process (performance may drop)
> [requested=29251MB, available=15942MB]
>
> Please tune the following settings:
> [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> MemoryConfiguration.systemCacheMaxSize = suggested value
> [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> value]
> [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> value]
>
> Current settings:
> [DefaultMemoryPolicySize = value]
> [{policy_name_1} size = value]
> [{policy_name_1} size = value]
> SystemCacheInitialSize = value
> SystemCacheMaxSize = value
> Java Heap Init Size = value
> Java Heap Max Size = value
>
> The overall memory usage by all Ignite nodes on the host: value
> -------------------------------------------
>
> Records in […] are optional. If custom memory policy is not set or the
> default memory policy is overridden the output will miss some of the rows.
>
> As for systemCacheMaxSize, it should be show ONLY if the parameter was set
> explicitly by user code. Otherwise, the platform should be wise enough to
> instantiate it properly depending on the host memory usage.
>
> —
> Denis
>
> > On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <gv...@gmail.com>
> wrote:
> >
> > The message without logging layout:
> >
> > Not enough memory for current process [required=29251MB,
> available=15942MB].
> > Please change MemoryConfiguration.systemCacheMaxSize and
> > MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
> > for each node.
> >
> > Current settings:
> > HeapInit=250MB
> > HeapMax=3543MB
> > DefaultMemoryPolicySize=12753MB
> > SystemCacheInitialSize=40MB
> > SystemCacheMaxSize=100MB
> >
> > Other ignite instanses on the server require: 12853MB
> >
> > I think it's make sense to describe what these numbers are consist of.
> >
> > We simple say which parameters have an impact on how much memory the
> > instance needs and their (parameters) actual values.
> >
> > Also we notice that more than one Ignite instance are ran on the server
> or
> > workstation and it also consumes memory.
> >
> > 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > dsetrakyan@apache.org> написал:
> >
> >> Igor, what is this flood of WARN messaging coming after the text? Are we
> >> really going to print this whole thing out?
> >>
> >> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <gvvinblade@gmail.com
> >
> >> wrote:
> >>
> >>> This message appears on topology change in case the available memory is
> >>> exceeded
> >>>
> >>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <gv...@gmail.com>:
> >>>
> >>>> An example of current impl:
> >>>>
> >>>>
> >>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> >>>> current process [required=29251MB, available=15942MB].
> >>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> >>>> MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> >>> defaultMemoryPolicySize
> >>>> to decrease memory allocated for each node.
> >>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> >>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
> >>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
> >>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> DefaultMemoryPolicySize=
> >>>> 12753MB
> >>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>> SystemCacheInitialSize=40MB
> >>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> SystemCacheMaxSize=100MB
> >>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses
> >> on
> >>>> the server require: 12853MB
> >>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> >>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >>>>
> >>>>
> >>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
> >>>>
> >>>>> I suppose we should not forget JVM heap size and suggest to reduce
> >> both
> >>>>> options
> >>>>>
> >>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> >>> dsetrakyan@apache.org
> >>>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Igor, I would change the message. How about this:
> >>>>>>
> >>>>>> Required RAM size is larger than total physical memory available for
> >>> OS.
> >>>>>>> Please change MemoryConfiguration.WhichProperty and
> >>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease memory
> >>> allocated
> >>>>> for
> >>>>>>> each node.
> >>>>>>
> >>>>>>
> >>>>>> Also, can we calculate what the memory size allocated for each node
> >>>>> should
> >>>>>> be? In that case we should suggest it.
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> >>> gvvinblade@gmail.com
> >>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> What do you, guys think about next warning?
> >>>>>>>
> >>>>>>> [2017-08-25 17:17:04,718][INFO
> >>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>> hSelfTest0]
> >>>>>>> System cache's MemoryPolicy size is configured to 40 MB. Use
> >>>>>>> MemoryConfiguration.systemCacheMemorySize property to change the
> >>>>>> setting.
> >>>>>>> [2017-08-25 17:17:04,718][WARN
> >>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>> hSelfTest0]
> >>>>>>>
> >>>>>>>>>> Required RAM size is larger than total physical memory
> >> available
> >>>>> for
> >>>>>>> OS.
> >>>>>>>>>> Check your configuration to avoid swap partition usage.
> >>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration to
> >> change
> >>>>> the
> >>>>>>> settings.
> >>>>>>>>>> Physical memory [required=16397MB, available=15942MB]
> >>>>>>>
> >>>>>>> [2017-08-25 17:17:04,726][WARN
> >>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>> hSelfTest0]
> >>>>>>> Peer class loading is enabled (disable it in production for
> >>>>> performance
> >>>>>> and
> >>>>>>> deployment consistency reasons)
> >>>>>>> [2017-08-25 17:17:04,726][INFO
> >>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>> hSelfTest0]
> >>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
> >>> ['ignite-sys-cache']]
> >>>>>>> [2017-08-25 17:17:04,731][INFO
> >>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >>>>> hSelfTest0]
> >>>>>>> 3-rd party licenses can be found at:
> >>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> >>>>>>>
> >>>>>>>
> >>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
> >>>>>>>
> >>>>>>>> Agree, let's release new version including tickets mentioned by
> >>>>> Denis
> >>>>>> and
> >>>>>>>> Ivan.
> >>>>>>>>
> >>>>>>>> --Yakov
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Sergey Kozlov
> >>>>> GridGain Systems
> >>>>> www.gridgain.com
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Denis Magda <dm...@apache.org>.
Igor,
Let me suggest this format.
---------------------------------------------
Excessive memory usage by Ignite node process (performance may drop) [requested=29251MB, available=15942MB]
Please tune the following settings:
[MemoryConfiguration.defaultMemoryPolicySize = suggested value]
MemoryConfiguration.systemCacheMaxSize = suggested value
[MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested value]
[MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested value]
Current settings:
[DefaultMemoryPolicySize = value]
[{policy_name_1} size = value]
[{policy_name_1} size = value]
SystemCacheInitialSize = value
SystemCacheMaxSize = value
Java Heap Init Size = value
Java Heap Max Size = value
The overall memory usage by all Ignite nodes on the host: value
-------------------------------------------
Records in […] are optional. If custom memory policy is not set or the default memory policy is overridden the output will miss some of the rows.
As for systemCacheMaxSize, it should be show ONLY if the parameter was set explicitly by user code. Otherwise, the platform should be wise enough to instantiate it properly depending on the host memory usage.
—
Denis
> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <gv...@gmail.com> wrote:
>
> The message without logging layout:
>
> Not enough memory for current process [required=29251MB, available=15942MB].
> Please change MemoryConfiguration.systemCacheMaxSize and
> MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
> for each node.
>
> Current settings:
> HeapInit=250MB
> HeapMax=3543MB
> DefaultMemoryPolicySize=12753MB
> SystemCacheInitialSize=40MB
> SystemCacheMaxSize=100MB
>
> Other ignite instanses on the server require: 12853MB
>
> I think it's make sense to describe what these numbers are consist of.
>
> We simple say which parameters have an impact on how much memory the
> instance needs and their (parameters) actual values.
>
> Also we notice that more than one Ignite instance are ran on the server or
> workstation and it also consumes memory.
>
> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> dsetrakyan@apache.org> написал:
>
>> Igor, what is this flood of WARN messaging coming after the text? Are we
>> really going to print this whole thing out?
>>
>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <gv...@gmail.com>
>> wrote:
>>
>>> This message appears on topology change in case the available memory is
>>> exceeded
>>>
>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <gv...@gmail.com>:
>>>
>>>> An example of current impl:
>>>>
>>>>
>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
>>>> current process [required=29251MB, available=15942MB].
>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
>>>> MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
>>> defaultMemoryPolicySize
>>>> to decrease memory allocated for each node.
>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> DefaultMemoryPolicySize=
>>>> 12753MB
>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>> SystemCacheInitialSize=40MB
>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>> SystemCacheMaxSize=100MB
>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses
>> on
>>>> the server require: 12853MB
>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
>>>>
>>>>
>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
>>>>
>>>>> I suppose we should not forget JVM heap size and suggest to reduce
>> both
>>>>> options
>>>>>
>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>> dsetrakyan@apache.org
>>>>>>
>>>>> wrote:
>>>>>
>>>>>> Igor, I would change the message. How about this:
>>>>>>
>>>>>> Required RAM size is larger than total physical memory available for
>>> OS.
>>>>>>> Please change MemoryConfiguration.WhichProperty and
>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease memory
>>> allocated
>>>>> for
>>>>>>> each node.
>>>>>>
>>>>>>
>>>>>> Also, can we calculate what the memory size allocated for each node
>>>>> should
>>>>>> be? In that case we should suggest it.
>>>>>>
>>>>>> D.
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
>>> gvvinblade@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> What do you, guys think about next warning?
>>>>>>>
>>>>>>> [2017-08-25 17:17:04,718][INFO
>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>> hSelfTest0]
>>>>>>> System cache's MemoryPolicy size is configured to 40 MB. Use
>>>>>>> MemoryConfiguration.systemCacheMemorySize property to change the
>>>>>> setting.
>>>>>>> [2017-08-25 17:17:04,718][WARN
>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>> hSelfTest0]
>>>>>>>
>>>>>>>>>> Required RAM size is larger than total physical memory
>> available
>>>>> for
>>>>>>> OS.
>>>>>>>>>> Check your configuration to avoid swap partition usage.
>>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration to
>> change
>>>>> the
>>>>>>> settings.
>>>>>>>>>> Physical memory [required=16397MB, available=15942MB]
>>>>>>>
>>>>>>> [2017-08-25 17:17:04,726][WARN
>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>> hSelfTest0]
>>>>>>> Peer class loading is enabled (disable it in production for
>>>>> performance
>>>>>> and
>>>>>>> deployment consistency reasons)
>>>>>>> [2017-08-25 17:17:04,726][INFO
>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>> hSelfTest0]
>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy:
>>> ['ignite-sys-cache']]
>>>>>>> [2017-08-25 17:17:04,731][INFO
>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>>>>> hSelfTest0]
>>>>>>> 3-rd party licenses can be found at:
>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
>>>>>>>
>>>>>>>
>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
>>>>>>>
>>>>>>>> Agree, let's release new version including tickets mentioned by
>>>>> Denis
>>>>>> and
>>>>>>>> Ivan.
>>>>>>>>
>>>>>>>> --Yakov
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sergey Kozlov
>>>>> GridGain Systems
>>>>> www.gridgain.com
>>>>>
>>>>
>>>>
>>>
>>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Seliverstov Igor <gv...@gmail.com>.
The message without logging layout:
Not enough memory for current process [required=29251MB, available=15942MB].
Please change MemoryConfiguration.systemCacheMaxSize and
MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
for each node.
Current settings:
HeapInit=250MB
HeapMax=3543MB
DefaultMemoryPolicySize=12753MB
SystemCacheInitialSize=40MB
SystemCacheMaxSize=100MB
Other ignite instanses on the server require: 12853MB
I think it's make sense to describe what these numbers are consist of.
We simple say which parameters have an impact on how much memory the
instance needs and their (parameters) actual values.
Also we notice that more than one Ignite instance are ran on the server or
workstation and it also consumes memory.
25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
dsetrakyan@apache.org> написал:
> Igor, what is this flood of WARN messaging coming after the text? Are we
> really going to print this whole thing out?
>
> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <gv...@gmail.com>
> wrote:
>
> > This message appears on topology change in case the available memory is
> > exceeded
> >
> > 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <gv...@gmail.com>:
> >
> > > An example of current impl:
> > >
> > >
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> > > current process [required=29251MB, available=15942MB].
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > > MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> > defaultMemoryPolicySize
> > > to decrease memory allocated for each node.
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> DefaultMemoryPolicySize=
> > > 12753MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > SystemCacheInitialSize=40MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> SystemCacheMaxSize=100MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses
> on
> > > the server require: 12853MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > >
> > >
> > > 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
> > >
> > >> I suppose we should not forget JVM heap size and suggest to reduce
> both
> > >> options
> > >>
> > >> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > dsetrakyan@apache.org
> > >> >
> > >> wrote:
> > >>
> > >> > Igor, I would change the message. How about this:
> > >> >
> > >> > Required RAM size is larger than total physical memory available for
> > OS.
> > >> > > Please change MemoryConfiguration.WhichProperty and
> > >> > > MemoryPolicyConfiguration.WhichProperty to decrease memory
> > allocated
> > >> for
> > >> > > each node.
> > >> >
> > >> >
> > >> > Also, can we calculate what the memory size allocated for each node
> > >> should
> > >> > be? In that case we should suggest it.
> > >> >
> > >> > D.
> > >> >
> > >> >
> > >> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > gvvinblade@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > What do you, guys think about next warning?
> > >> > >
> > >> > > [2017-08-25 17:17:04,718][INFO
> > >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >> hSelfTest0]
> > >> > > System cache's MemoryPolicy size is configured to 40 MB. Use
> > >> > > MemoryConfiguration.systemCacheMemorySize property to change the
> > >> > setting.
> > >> > > [2017-08-25 17:17:04,718][WARN
> > >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >> hSelfTest0]
> > >> > >
> > >> > > >>> Required RAM size is larger than total physical memory
> available
> > >> for
> > >> > > OS.
> > >> > > >>> Check your configuration to avoid swap partition usage.
> > >> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to
> change
> > >> the
> > >> > > settings.
> > >> > > >>> Physical memory [required=16397MB, available=15942MB]
> > >> > >
> > >> > > [2017-08-25 17:17:04,726][WARN
> > >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >> hSelfTest0]
> > >> > > Peer class loading is enabled (disable it in production for
> > >> performance
> > >> > and
> > >> > > deployment consistency reasons)
> > >> > > [2017-08-25 17:17:04,726][INFO
> > >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >> hSelfTest0]
> > >> > > Configured caches [in 'sysMemPlc' memoryPolicy:
> > ['ignite-sys-cache']]
> > >> > > [2017-08-25 17:17:04,731][INFO
> > >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >> hSelfTest0]
> > >> > > 3-rd party licenses can be found at:
> > >> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> > >> > >
> > >> > >
> > >> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
> > >> > >
> > >> > > > Agree, let's release new version including tickets mentioned by
> > >> Denis
> > >> > and
> > >> > > > Ivan.
> > >> > > >
> > >> > > > --Yakov
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Sergey Kozlov
> > >> GridGain Systems
> > >> www.gridgain.com
> > >>
> > >
> > >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Igor, what is this flood of WARN messaging coming after the text? Are we
really going to print this whole thing out?
On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <gv...@gmail.com>
wrote:
> This message appears on topology change in case the available memory is
> exceeded
>
> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor <gv...@gmail.com>:
>
> > An example of current impl:
> >
> >
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> > current process [required=29251MB, available=15942MB].
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> defaultMemoryPolicySize
> > to decrease memory allocated for each node.
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] DefaultMemoryPolicySize=
> > 12753MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> > SystemCacheInitialSize=40MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] SystemCacheMaxSize=100MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses on
> > the server require: 12853MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> >
> >
> > 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
> >
> >> I suppose we should not forget JVM heap size and suggest to reduce both
> >> options
> >>
> >> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> >> >
> >> wrote:
> >>
> >> > Igor, I would change the message. How about this:
> >> >
> >> > Required RAM size is larger than total physical memory available for
> OS.
> >> > > Please change MemoryConfiguration.WhichProperty and
> >> > > MemoryPolicyConfiguration.WhichProperty to decrease memory
> allocated
> >> for
> >> > > each node.
> >> >
> >> >
> >> > Also, can we calculate what the memory size allocated for each node
> >> should
> >> > be? In that case we should suggest it.
> >> >
> >> > D.
> >> >
> >> >
> >> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> gvvinblade@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > What do you, guys think about next warning?
> >> > >
> >> > > [2017-08-25 17:17:04,718][INFO
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > > System cache's MemoryPolicy size is configured to 40 MB. Use
> >> > > MemoryConfiguration.systemCacheMemorySize property to change the
> >> > setting.
> >> > > [2017-08-25 17:17:04,718][WARN
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > >
> >> > > >>> Required RAM size is larger than total physical memory available
> >> for
> >> > > OS.
> >> > > >>> Check your configuration to avoid swap partition usage.
> >> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change
> >> the
> >> > > settings.
> >> > > >>> Physical memory [required=16397MB, available=15942MB]
> >> > >
> >> > > [2017-08-25 17:17:04,726][WARN
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > > Peer class loading is enabled (disable it in production for
> >> performance
> >> > and
> >> > > deployment consistency reasons)
> >> > > [2017-08-25 17:17:04,726][INFO
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > > Configured caches [in 'sysMemPlc' memoryPolicy:
> ['ignite-sys-cache']]
> >> > > [2017-08-25 17:17:04,731][INFO
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > > 3-rd party licenses can be found at:
> >> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> >> > >
> >> > >
> >> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
> >> > >
> >> > > > Agree, let's release new version including tickets mentioned by
> >> Denis
> >> > and
> >> > > > Ivan.
> >> > > >
> >> > > > --Yakov
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Sergey Kozlov
> >> GridGain Systems
> >> www.gridgain.com
> >>
> >
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Seliverstov Igor <gv...@gmail.com>.
This message appears on topology change in case the available memory is
exceeded
2017-08-25 19:47 GMT+03:00 Seliverstov Igor <gv...@gmail.com>:
> An example of current impl:
>
>
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> current process [required=29251MB, available=15942MB].
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.defaultMemoryPolicySize
> to decrease memory allocated for each node.
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] HeapInit=250MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] HeapMax=3543MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] DefaultMemoryPolicySize=
> 12753MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> SystemCacheInitialSize=40MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] SystemCacheMaxSize=100MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses on
> the server require: 12853MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
>
>
> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
>
>> I suppose we should not forget JVM heap size and suggest to reduce both
>> options
>>
>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
>> >
>> wrote:
>>
>> > Igor, I would change the message. How about this:
>> >
>> > Required RAM size is larger than total physical memory available for OS.
>> > > Please change MemoryConfiguration.WhichProperty and
>> > > MemoryPolicyConfiguration.WhichProperty to decrease memory allocated
>> for
>> > > each node.
>> >
>> >
>> > Also, can we calculate what the memory size allocated for each node
>> should
>> > be? In that case we should suggest it.
>> >
>> > D.
>> >
>> >
>> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <gvvinblade@gmail.com
>> >
>> > wrote:
>> >
>> > > What do you, guys think about next warning?
>> > >
>> > > [2017-08-25 17:17:04,718][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > System cache's MemoryPolicy size is configured to 40 MB. Use
>> > > MemoryConfiguration.systemCacheMemorySize property to change the
>> > setting.
>> > > [2017-08-25 17:17:04,718][WARN
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > >
>> > > >>> Required RAM size is larger than total physical memory available
>> for
>> > > OS.
>> > > >>> Check your configuration to avoid swap partition usage.
>> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change
>> the
>> > > settings.
>> > > >>> Physical memory [required=16397MB, available=15942MB]
>> > >
>> > > [2017-08-25 17:17:04,726][WARN
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > Peer class loading is enabled (disable it in production for
>> performance
>> > and
>> > > deployment consistency reasons)
>> > > [2017-08-25 17:17:04,726][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
>> > > [2017-08-25 17:17:04,731][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > 3-rd party licenses can be found at:
>> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
>> > >
>> > >
>> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
>> > >
>> > > > Agree, let's release new version including tickets mentioned by
>> Denis
>> > and
>> > > > Ivan.
>> > > >
>> > > > --Yakov
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Sergey Kozlov
>> GridGain Systems
>> www.gridgain.com
>>
>
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Seliverstov Igor <gv...@gmail.com>.
An example of current impl:
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Not enough memory for current process [required=29251MB, available=15942MB].
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Please change MemoryConfiguration.systemCacheMaxSize and
MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
for each node.
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Current settings:
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
HeapInit=250MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
HeapMax=3543MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
DefaultMemoryPolicySize=12753MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
SystemCacheInitialSize=40MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
SystemCacheMaxSize=100MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Other ignite instanses on the server require: 12853MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
2017-08-25 17:40 GMT+03:00 Sergey Kozlov <sk...@gridgain.com>:
> I suppose we should not forget JVM heap size and suggest to reduce both
> options
>
> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > Igor, I would change the message. How about this:
> >
> > Required RAM size is larger than total physical memory available for OS.
> > > Please change MemoryConfiguration.WhichProperty and
> > > MemoryPolicyConfiguration.WhichProperty to decrease memory allocated
> for
> > > each node.
> >
> >
> > Also, can we calculate what the memory size allocated for each node
> should
> > be? In that case we should suggest it.
> >
> > D.
> >
> >
> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <gv...@gmail.com>
> > wrote:
> >
> > > What do you, guys think about next warning?
> > >
> > > [2017-08-25 17:17:04,718][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > System cache's MemoryPolicy size is configured to 40 MB. Use
> > > MemoryConfiguration.systemCacheMemorySize property to change the
> > setting.
> > > [2017-08-25 17:17:04,718][WARN
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > >
> > > >>> Required RAM size is larger than total physical memory available
> for
> > > OS.
> > > >>> Check your configuration to avoid swap partition usage.
> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
> > > settings.
> > > >>> Physical memory [required=16397MB, available=15942MB]
> > >
> > > [2017-08-25 17:17:04,726][WARN
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > Peer class loading is enabled (disable it in production for performance
> > and
> > > deployment consistency reasons)
> > > [2017-08-25 17:17:04,726][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
> > > [2017-08-25 17:17:04,731][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > 3-rd party licenses can be found at:
> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> > >
> > >
> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
> > >
> > > > Agree, let's release new version including tickets mentioned by Denis
> > and
> > > > Ivan.
> > > >
> > > > --Yakov
> > > >
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Sergey Kozlov <sk...@gridgain.com>.
I suppose we should not forget JVM heap size and suggest to reduce both
options
On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Igor, I would change the message. How about this:
>
> Required RAM size is larger than total physical memory available for OS.
> > Please change MemoryConfiguration.WhichProperty and
> > MemoryPolicyConfiguration.WhichProperty to decrease memory allocated for
> > each node.
>
>
> Also, can we calculate what the memory size allocated for each node should
> be? In that case we should suggest it.
>
> D.
>
>
> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <gv...@gmail.com>
> wrote:
>
> > What do you, guys think about next warning?
> >
> > [2017-08-25 17:17:04,718][INFO
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> > System cache's MemoryPolicy size is configured to 40 MB. Use
> > MemoryConfiguration.systemCacheMemorySize property to change the
> setting.
> > [2017-08-25 17:17:04,718][WARN
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> >
> > >>> Required RAM size is larger than total physical memory available for
> > OS.
> > >>> Check your configuration to avoid swap partition usage.
> > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
> > settings.
> > >>> Physical memory [required=16397MB, available=15942MB]
> >
> > [2017-08-25 17:17:04,726][WARN
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> > Peer class loading is enabled (disable it in production for performance
> and
> > deployment consistency reasons)
> > [2017-08-25 17:17:04,726][INFO
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> > Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
> > [2017-08-25 17:17:04,731][INFO
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> > 3-rd party licenses can be found at:
> > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> >
> >
> > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
> >
> > > Agree, let's release new version including tickets mentioned by Denis
> and
> > > Ivan.
> > >
> > > --Yakov
> > >
> >
>
--
Sergey Kozlov
GridGain Systems
www.gridgain.com
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Igor, I would change the message. How about this:
Required RAM size is larger than total physical memory available for OS.
> Please change MemoryConfiguration.WhichProperty and
> MemoryPolicyConfiguration.WhichProperty to decrease memory allocated for
> each node.
Also, can we calculate what the memory size allocated for each node should
be? In that case we should suggest it.
D.
On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <gv...@gmail.com>
wrote:
> What do you, guys think about next warning?
>
> [2017-08-25 17:17:04,718][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> System cache's MemoryPolicy size is configured to 40 MB. Use
> MemoryConfiguration.systemCacheMemorySize property to change the setting.
> [2017-08-25 17:17:04,718][WARN
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
>
> >>> Required RAM size is larger than total physical memory available for
> OS.
> >>> Check your configuration to avoid swap partition usage.
> >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
> settings.
> >>> Physical memory [required=16397MB, available=15942MB]
>
> [2017-08-25 17:17:04,726][WARN
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> Peer class loading is enabled (disable it in production for performance and
> deployment consistency reasons)
> [2017-08-25 17:17:04,726][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
> [2017-08-25 17:17:04,731][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> 3-rd party licenses can be found at:
> /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
>
>
> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
>
> > Agree, let's release new version including tickets mentioned by Denis and
> > Ivan.
> >
> > --Yakov
> >
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Seliverstov Igor <gv...@gmail.com>.
What do you, guys think about next warning?
[2017-08-25 17:17:04,718][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
System cache's MemoryPolicy size is configured to 40 MB. Use
MemoryConfiguration.systemCacheMemorySize property to change the setting.
[2017-08-25 17:17:04,718][WARN
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
>>> Required RAM size is larger than total physical memory available for OS.
>>> Check your configuration to avoid swap partition usage.
>>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
settings.
>>> Physical memory [required=16397MB, available=15942MB]
[2017-08-25 17:17:04,726][WARN
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
Peer class loading is enabled (disable it in production for performance and
deployment consistency reasons)
[2017-08-25 17:17:04,726][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
[2017-08-25 17:17:04,731][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
3-rd party licenses can be found at:
/home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
2017-08-25 13:26 GMT+03:00 Yakov Zhdanov <yz...@apache.org>:
> Agree, let's release new version including tickets mentioned by Denis and
> Ivan.
>
> --Yakov
>
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Yakov Zhdanov <yz...@apache.org>.
Agree, let's release new version including tickets mentioned by Denis and
Ivan.
--Yakov
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Ivan Rakov <iv...@gmail.com>.
Let's include to the release tickets with optimizations of checkpointing
algorithm:
https://issues.apache.org/jira/browse/IGNITE-6178
https://issues.apache.org/jira/browse/IGNITE-6033
https://issues.apache.org/jira/browse/IGNITE-5961
This will help users who are experiencing problems with slow checkpoints.
Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to
soften "Ignite node crashed in the middle of checkpoint" message as per
discussion
<http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html>.
Best Regards,
Ivan Rakov
On 25.08.2017 3:31, Dmitriy Setrakyan wrote:
> +1 on the release, let's do it ASAP.
>
> On Thu, Aug 24, 2017 at 12:32 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Igniters,
>>
>> Presently every Apache Ignite node requests 80% of RAM from an operating
>> system. We debated a lot about this default [1] and as a practice shows
>> those who voted for the lower value (20%-25%) were right.
>>
>> Our users started reporting about bizarre issues ([2], [3], [4]) when a
>> cluster hangs, freezes for a while or even crashes during standard
>> preloading phase or minor workloads.
>>
>> It’s turned out to be that lack of RAM causes extensive swapping and
>> checkpointing activity. Even more, the issue can be reproduced on your
>> laptop if you launch a couple of nodes and start data preloading.
>>
>> All this can be avoided if you allocated less than 80% and tune the
>> checkpointing if the persistence is used. Unfortunately, only few could get
>> to this point, the rest would kick Ignite out from their deployments.
>>
>> Thus, we have to fix the defaults ([5] and [6]) on our own and release an
>> emergency release with the fixes as soon as possible.
>>
>> Any objections or other thoughts?
>>
>> [1] http://apache-ignite-developers.2346864.n4.nabble.
>> com/IGNITE-5717-improvements-of-MemoryPolicy-default-size-td20264.html
>> [2] http://apache-ignite-users.70518.x6.nabble.com/
>> Performance-of-persistent-store-too-low-when-bulb-
>> loading-tp16247p16337.html
>> [3] http://apache-ignite-users.70518.x6.nabble.com/Strange-
>> problems-with-Ignite-native-Persistence-when-Data-exceeds-
>> Memory-td16187.html#a16194
>> [4] http://apache-ignite-users.70518.x6.nabble.com/
>> Activation-slow-and-Ignite-node-crashed-in-the-middle-of-
>> checkpoint-td16144.html
>>
>> [5] https://issues.apache.org/jira/browse/IGNITE-6003
>> [6] https://issues.apache.org/jira/browse/IGNITE-6182
>>
>> —
>> Denis
Re: [DISCUSSION] Urgent Ignite bug fix release
Posted by Dmitriy Setrakyan <ds...@apache.org>.
+1 on the release, let's do it ASAP.
On Thu, Aug 24, 2017 at 12:32 PM, Denis Magda <dm...@apache.org> wrote:
> Igniters,
>
> Presently every Apache Ignite node requests 80% of RAM from an operating
> system. We debated a lot about this default [1] and as a practice shows
> those who voted for the lower value (20%-25%) were right.
>
> Our users started reporting about bizarre issues ([2], [3], [4]) when a
> cluster hangs, freezes for a while or even crashes during standard
> preloading phase or minor workloads.
>
> It’s turned out to be that lack of RAM causes extensive swapping and
> checkpointing activity. Even more, the issue can be reproduced on your
> laptop if you launch a couple of nodes and start data preloading.
>
> All this can be avoided if you allocated less than 80% and tune the
> checkpointing if the persistence is used. Unfortunately, only few could get
> to this point, the rest would kick Ignite out from their deployments.
>
> Thus, we have to fix the defaults ([5] and [6]) on our own and release an
> emergency release with the fixes as soon as possible.
>
> Any objections or other thoughts?
>
> [1] http://apache-ignite-developers.2346864.n4.nabble.
> com/IGNITE-5717-improvements-of-MemoryPolicy-default-size-td20264.html
> [2] http://apache-ignite-users.70518.x6.nabble.com/
> Performance-of-persistent-store-too-low-when-bulb-
> loading-tp16247p16337.html
> [3] http://apache-ignite-users.70518.x6.nabble.com/Strange-
> problems-with-Ignite-native-Persistence-when-Data-exceeds-
> Memory-td16187.html#a16194
> [4] http://apache-ignite-users.70518.x6.nabble.com/
> Activation-slow-and-Ignite-node-crashed-in-the-middle-of-
> checkpoint-td16144.html
>
> [5] https://issues.apache.org/jira/browse/IGNITE-6003
> [6] https://issues.apache.org/jira/browse/IGNITE-6182
>
> —
> Denis