AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

[chris_d <ch...@fsbtech.com>]

Hi,

I have a question/comment about the behaviour of adaptive load balancing
with Ignite 2.7.

I was running a load test on our systems and noticed that eventually we were
running out of heap space. Most of the heap was being taken up by the
taskTops map within AdaptiveLoadBalancingSpi.

Each new task was adding topology data to that map but nothing was ever
getting removed.

There is a GridLocalEventListener registered in AdaptiveLoadBalancingSpi
that will remove tasks from the map if events of type
EVT_TASK_FINISHED/EVT_TASK_FAILED are received.

The problem is that those event types don't seem to be recorded by default.

It seems easy enough to sort by using
IgniteConfiguration.setIncludeEventTypes(EventType.EVT_TASK_FINISHED,
EventType.EVT_TASK_FAILED) but it's not obvious that this is necessary in
the first place. It took me a fair bit of digging through the code and
debugging to work out what was going on.

Is it possible to improve the adaptive load balancing behaviour a bit?
Ensuring the finished/failed event tasks get recorded by default when
adaptive load balancing is used perhaps?

Thanks,
Chris.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

[Andrei Aleksandrov <ae...@gmail.com>]

Hi,

I tested your case and looks like Ignite have the memory issue here.

I filed the ticket for it:

https://issues.apache.org/jira/browse/IGNITE-11966

BR,
Andrei

On 2019/06/03 13:04:49, chris_d <c....@fsbtech.com> wrote:
 > Hi Andrei,>
 >
 > I've attached a zip of the top consumers and the class we're using to>
 > configure the ignite clients within our app server.>
 >
 > I can't really provide all the code involved as the load test was 
testing>
 > quite a large chunk of our system.>
 >
 > If necessary I can try and create a cut-down version of the test. >
 >
 > 855_Top_Consumers.zip>
 > 
<http://apache-ignite-users.70518.x6.nabble.com/file/t2459/855_Top_Consumers.zip> 
 >
 >
 > AbstractGridConfigurationBuilder.java>
 > 
<http://apache-ignite-users.70518.x6.nabble.com/file/t2459/AbstractGridConfigurationBuilder.java> 
 >
 >
 > Thanks>
 > Chris.>
 >
 >
 >
 > -->
 > Sent from: http://apache-ignite-users.70518.x6.nabble.com/>
 >

Re: AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

[chris_d <ch...@fsbtech.com>]

Hi Andrei,

I've attached a zip of the top consumers and the class we're using to
configure the ignite clients within our app server.

I can't really provide all the code involved as the load test was testing
quite a large chunk of our system.

If necessary I can try and create a cut-down version of the test. 

855_Top_Consumers.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t2459/855_Top_Consumers.zip>  

AbstractGridConfigurationBuilder.java
<http://apache-ignite-users.70518.x6.nabble.com/file/t2459/AbstractGridConfigurationBuilder.java>  

Thanks
Chris.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: AdaptiveLoadBalancingSpi not removing finished tasks from taskTops map

[aealexsandrov <ae...@gmail.com>]

Hi,

It looks like an issue that possible can be a reason for OOM exception.

Is it possible to share the top consumers class list from the heap dump (it
can be done via memory analyzer e.g Eclipse MAT)?

Also, is it possible to provide the code and configuration that was used in
your load testing?

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/