You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Nirmal Fernando <ni...@gmail.com> on 2014/11/05 21:02:59 UTC
[Fixed] CEP sends very large values for gradient and second
derivative of load average
Hi Guys,
So, I got a chance to reproduce and analyze this issue.
How we calculate gradient of two events?
Say the events are; e1(t1,v1) and e2(t2,v2)
tx - time in *milliseconds* when xth event occurred
vx - value (memory, cpu etc.) that xth event carries
*time gap = t(2-1) = t2 -t1 milliseconds*
time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
Hence,
*Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
I've enabled debug logs for CEP extension;
log4j.logger.org.apache.stratos.cep.extension=DEBUG
Please find the following 3 logs extracted from the debug logs;
===================================================================
TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
{org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
{org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor}
- Gradient:
-999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time Gap: 5*
t1: 1415213232152 t2: 1415213232157 hash: 155426542
TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
{org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
-0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2: 1415213232154
hash: 155426542
===================================================================
So, as you can see the reason behind a large value is when the time gap
between two subjected events is less than 1 second. This could happen since
events are coming from different asynchronous agents and also when there
are less number of events.
So, the fix I propose is a very simple one and it will not compromise
anything AFAIS.
Fix is to calculate time gap as follows;
*____ t2 - t1*
*| yes?*
*time gap = t(2-1) = t2 -t1 > 1000 ---*
*|____ 1000*
I have tested this and works fine.
---------- Forwarded message ----------
From: Manula Chathurika Thantriwatte <ma...@wso2.com>
Date: Tue, Oct 28, 2014 at 5:16 AM
Subject: Re: CEP sends very large values for gradient and second derivative
of load average
To: "dev@stratos.apache.org" <de...@stratos.apache.org>
Hi Raj,
Yes it has. But it's very easy to find the values from the agent with the
time stamp. Then we can search that time stamp in the CEP trace logs and
compare the summarized values.
Thanks !
On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
>
>
> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
> manulac@wso2.com> wrote:
>
>> Hi Raj,
>>
>> Shall we enable the cartridge agent debug logs. Then we can compare both
>> CEP and agent logs with the time stamp. Then we can narrow down where the
>> actual problem is.
>>
>
> Manula, cep-trace log have the values sent by cartridge agent also right?
>
>
>>
>> Thanks !
>>
>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>> manulac@wso2.com> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> Is this happen frequently or intermittently ? Once I go though the logs
>>>> I'm unable to find the high input values for load average. But output have
>>>> high values.
>>>>
>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>> JMSOutputAdaptor, sent
>>>>
>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>
>>>
>>> This is happening intermittently. Experienced now too. This is a serious
>>> issue. We need to somehow sort this out.
>>>
>>>
>>>>
>>>> Also networkPartitionId is empty. In the current execution plans it
>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>> execution plans for 4.1.0.
>>>>
>>>> Thanks !
>>>>
>>>>
>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>> manulac@wso2.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thanks Raj. I'll go through them.
>>>>>
>>>>> Thanks !
>>>>>
>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>> Hi Manula,
>>>>>>
>>>>>> You can see those values in the same log I attached in the previous
>>>>>> mail.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Chamila,
>>>>>>>
>>>>>>> I couldn't find any large values sent by agent. There are traces for
>>>>>>> CEP sending large values. Find the cep trace log here [1]
>>>>>>>
>>>>>>> 1.
>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>> manulac@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Shall we enable the cartridge agent debug logs and see what are the
>>>>>>>> load average values sent from the agent. If those values are normal then we
>>>>>>>> can narrow down to the CEP.
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>>
>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Raj,
>>>>>>>>>
>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>> will have the events published from the agent.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Chamila de Alwis
>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am doing a test round with M3 with python agent. And faced this
>>>>>>>>>> very strange scenario, $Subject.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>> 200000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>> check executed for :
>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>> loadAverageReset : true
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>> intervals]: 1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>> intervals]: 1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>> destroyed.
>>>>>>>>>>
>>>>>>>>>> Any idea?
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>> phone : +94 772492511
>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rajkumar Rajaratnam
>>>>>> Software Engineer | WSO2, Inc.
>>>>>> Mobile +94777568639 | +94783498120
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manulac@wso2.com / manula@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>
>
> --
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
> Mobile +94777568639 | +94783498120
>
--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware
email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
On Thu, Nov 6, 2014 at 3:47 AM, Akila Ravihansa Perera <ra...@wso2.com>
wrote:
> Hi Nirmal,
>
> Great finding!
>
> But I don't follow your reasoning for this behavior. Could you explain how
> this can be related to "less number of events" being published by agents?
>
> IMHO, events coming from agents should be sampled and averaged within a
> fixed time intervals. This should be done in a separate CEP execution plan.
> Those sampled events then should be sent to SecondDerivativeFinderWindowProcessor.
> In that way we won't run into issues where time intervals are less than
> 1000 since it will take those values for pre-defined intervals.
>
> I might be missing a key piece here, but thought of sharing this :)
>
Thanks Akila! I think what you are missing here is, what we take into
account is the real timestamp values of the events. Imagine scenario where
we are calculating gradient for a cluster which spans few members. The
agents of those members are sending events in a desynchronized manner. If
you take a 60second interval and if you take the events by timestamp, you
would see a graph like below;
|*.. . . .... ... . * |
In this case, for a more accurate value, currently we do not assume time
gap is 60s, instead we take the real timestamp differences of the first and
last event. This is a very simple calculation of gradient, and I think it's
sufficient for our needs.
Then in order to find second derivative; we would select another two events
if possible.
|*.. . . .... ... . * |
this is a two step process, where we would calculate gradient from red
dotted events and green dotted events separately and get the timestamp of
the new gradient values as (t1+t2)/2. Then we again find the gradient from
the two new gradient events and return it as the second derivative value.
So, when there're less number of events and when it occurs in a sequence
like below, we would need to normalize before calculating gradient.(t(e3) -
t(e2) < 1s)
|*. ..* |
> Thanks.
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Hi Nirmal,
Great finding!
But I don't follow your reasoning for this behavior. Could you explain how
this can be related to "less number of events" being published by agents?
IMHO, events coming from agents should be sampled and averaged within a
fixed time intervals. This should be done in a separate CEP execution plan.
Those sampled events then should be sent to
SecondDerivativeFinderWindowProcessor.
In that way we won't run into issues where time intervals are less than
1000 since it will take those values for pre-defined intervals.
I might be missing a key piece here, but thought of sharing this :)
Thanks.
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
Please find the Jira https://issues.apache.org/jira/browse/STRATOS-939
I've committed the fix.
On Wed, Nov 5, 2014 at 9:02 PM, Nirmal Fernando <ni...@gmail.com>
wrote:
> Hi Guys,
>
> So, I got a chance to reproduce and analyze this issue.
>
> How we calculate gradient of two events?
>
> Say the events are; e1(t1,v1) and e2(t2,v2)
>
> tx - time in *milliseconds* when xth event occurred
> vx - value (memory, cpu etc.) that xth event carries
>
> *time gap = t(2-1) = t2 -t1 milliseconds*
> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>
> Hence,
> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>
> I've enabled debug logs for CEP extension;
>
> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>
> Please find the following 3 logs extracted from the debug logs;
>
> ===================================================================
> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
> 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>
> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time Gap:
> 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>
> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2: 1415213232154
> hash: 155426542
> ===================================================================
>
> So, as you can see the reason behind a large value is when the time gap
> between two subjected events is less than 1 second. This could happen since
> events are coming from different asynchronous agents and also when there
> are less number of events.
>
> So, the fix I propose is a very simple one and it will not compromise
> anything AFAIS.
>
> Fix is to calculate time gap as follows;
> *____ t2 - t1*
> *| yes?*
> *time gap = t(2-1) = t2 -t1 > 1000 ---*
> *|____ 1000*
>
> I have tested this and works fine.
>
>
>
> ---------- Forwarded message ----------
> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
> Date: Tue, Oct 28, 2014 at 5:16 AM
> Subject: Re: CEP sends very large values for gradient and second
> derivative of load average
> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>
>
> Hi Raj,
>
> Yes it has. But it's very easy to find the values from the agent with the
> time stamp. Then we can search that time stamp in the CEP trace logs and
> compare the summarized values.
>
> Thanks !
>
> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>>
>>
>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>> manulac@wso2.com> wrote:
>>
>>> Hi Raj,
>>>
>>> Shall we enable the cartridge agent debug logs. Then we can compare both
>>> CEP and agent logs with the time stamp. Then we can narrow down where the
>>> actual problem is.
>>>
>>
>> Manula, cep-trace log have the values sent by cartridge agent also right?
>>
>>
>>>
>>> Thanks !
>>>
>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>> manulac@wso2.com> wrote:
>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>> logs I'm unable to find the high input values for load average. But output
>>>>> have high values.
>>>>>
>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>> JMSOutputAdaptor, sent
>>>>>
>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>
>>>>
>>>> This is happening intermittently. Experienced now too. This is a
>>>> serious issue. We need to somehow sort this out.
>>>>
>>>>
>>>>>
>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>>> execution plans for 4.1.0.
>>>>>
>>>>> Thanks !
>>>>>
>>>>>
>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>> manulac@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks Raj. I'll go through them.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Manula,
>>>>>>>
>>>>>>> You can see those values in the same log I attached in the previous
>>>>>>> mail.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Chamila,
>>>>>>>>
>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>> the load average values sent from the agent. If those values are normal
>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raj,
>>>>>>>>>>
>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>>> will have the events published from the agent.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Chamila de Alwis
>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>> 200000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>> check executed for :
>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>> intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>> intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>> destroyed.
>>>>>>>>>>>
>>>>>>>>>>> Any idea?
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Software Engineer | WSO2, Inc.
>>>> Mobile +94777568639 | +94783498120
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manula Chathurika Thantriwatte
>>> Software Engineer
>>> WSO2 Inc. : http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> email : manulac@wso2.com / manula@apache.org
>>> phone : +94 772492511
>>> blog : http://manulachathurika.blogspot.com/
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Software Engineer | WSO2, Inc.
>> Mobile +94777568639 | +94783498120
>>
>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manulac@wso2.com / manula@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
>
>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
On Thu, Nov 6, 2014 at 4:33 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Nirmal,
>
> I do not think the milliseconds to seconds convertion is correct here.
>
> As I see we are taking the difference of two timestamp values and then
> dividing it by 1000. The corect way might be to first divide each value by
> 1000 and then take the difference.
>
(5-1)/2 = 5/2 - 1/2 is the Math I know :-)
>
> On the other hand we might not need to convert these values to seconds
> since we are taking a time difference and calculating a gradient.
>
It's a time difference, so the value still matters, if it's division of
same units, then it can be neglected.
>
> I did a quick test with the following sample:
> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>
> *According previous code:*
> long tGap = t2 - t1;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * 1000) / tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -1000.0
>
> *According to your fix:*
> long millisecondsForASecond = 1000;
> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
> millisecondsForASecond;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
> tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -5.0
>
> *According to an online gradient calculator:*
> gradient: -1
>
> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>
> According to the online gradient calculator (assuming their calculation is
> correct), the calculation in your fix is not correct. I believe the logic
> should be simple as follows:
>
> long tGap = t2 - t1;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal)) / tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -1.0
>
This is not per second value.
So, what I've done here is to put a reasonable constraint on the events
saying, two events should be at least 1s apart from each other before doing
gradient (per second) calculation.
And this is the tangent value we're talking about and tangent can span from
0 to (+/-)infinity.
>
> Thanks
>
>
> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
>> Hi Guys,
>>
>> So, I got a chance to reproduce and analyze this issue.
>>
>> How we calculate gradient of two events?
>>
>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>
>> tx - time in *milliseconds* when xth event occurred
>> vx - value (memory, cpu etc.) that xth event carries
>>
>> *time gap = t(2-1) = t2 -t1 milliseconds*
>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>
>> Hence,
>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>
>> I've enabled debug logs for CEP extension;
>>
>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>
>> Please find the following 3 logs extracted from the debug logs;
>>
>> ===================================================================
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
>> 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>> 1415213232154 hash: 155426542
>> ===================================================================
>>
>> So, as you can see the reason behind a large value is when the time gap
>> between two subjected events is less than 1 second. This could happen since
>> events are coming from different asynchronous agents and also when there
>> are less number of events.
>>
>> So, the fix I propose is a very simple one and it will not compromise
>> anything AFAIS.
>>
>> Fix is to calculate time gap as follows;
>> *____ t2 - t1*
>> *| yes?*
>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>> *|____ 1000*
>>
>> I have tested this and works fine.
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>> Date: Tue, Oct 28, 2014 at 5:16 AM
>> Subject: Re: CEP sends very large values for gradient and second
>> derivative of load average
>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>
>>
>> Hi Raj,
>>
>> Yes it has. But it's very easy to find the values from the agent with the
>> time stamp. Then we can search that time stamp in the CEP trace logs and
>> compare the summarized values.
>>
>> Thanks !
>>
>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>> manulac@wso2.com> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>> the actual problem is.
>>>>
>>>
>>> Manula, cep-trace log have the values sent by cartridge agent also
>>> right?
>>>
>>>
>>>>
>>>> Thanks !
>>>>
>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>> manulac@wso2.com> wrote:
>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>>> logs I'm unable to find the high input values for load average. But output
>>>>>> have high values.
>>>>>>
>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>> JMSOutputAdaptor, sent
>>>>>>
>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>
>>>>>
>>>>> This is happening intermittently. Experienced now too. This is a
>>>>> serious issue. We need to somehow sort this out.
>>>>>
>>>>>
>>>>>>
>>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>>>> execution plans for 4.1.0.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>> manulac@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Manula,
>>>>>>>>
>>>>>>>> You can see those values in the same log I attached in the previous
>>>>>>>> mail.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Chamila,
>>>>>>>>>
>>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>
>>>>>>>>> 1.
>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>>> the load average values sent from the agent. If those values are normal
>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>
>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>>>> will have the events published from the agent.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 200000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>> check executed for :
>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>
>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Software Engineer | WSO2, Inc.
>>>>> Mobile +94777568639 | +94783498120
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manulac@wso2.com / manula@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Lahiru Sandaruwan <la...@wso2.com>.
Raj, Imesh and i had a discussion on an improvement to window size we
accumulate the stats. Currently we get 1 min window and run the execution
plan 1 min by 1 min. We can improve the sample by increasing the window to
10 mins and running it 1 min by 1 min.
I will incorporate this with the curve fitting change.
On Fri, Nov 7, 2014 at 12:47 AM, Akila Ravihansa Perera <ra...@wso2.com>
wrote:
> Hi,
>
> On Nov 6, 2014 11:37 AM, "Imesh Gunaratne" <im...@apache.org> wrote:
> >
> > Thanks for your response Nirmal, please see my thoughts below:
> >
> > On Thu, Nov 6, 2014 at 7:38 PM, Nirmal Fernando <nirmal070125@gmail.com
> > wrote:
> >>
> >> AFAIU if it is statistics, it's all about random data, samples and
> normalization. You don't use all values to do estimations. And this is an
> estimation for gradient per say!
> >
> >
> > True, however the random data needs to be accurate as much as possible.
> >>>
> >>>
> >> Well, statistics we are calculating is for a cluster as a whole not
> member wise. Since, we autoscale a cluster.
> >
> >
> > Yes for autoscaling a cluster the aggregated statistics should be
> calculated against the cluster. However I do not think that we can mix each
> statistic accorss members when calculating differences. Different members
> of a cluster might be running at different resource usage levels at a given
> point of time. Therefore aggregation might needed to be done at the member
> level first and then on the cluster level. WDYT?
>
> +1
>
> We need to sample stats and aggregate them on fixed intervals before
> calculating the gradients.
>
> Thanks.
>
> >
> >
> > --
> > Imesh Gunaratne
> >
> > Technical Lead, WSO2
> > Committer & PMC Member, Apache Stratos
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Hi,
On Nov 6, 2014 11:37 AM, "Imesh Gunaratne" <im...@apache.org> wrote:
>
> Thanks for your response Nirmal, please see my thoughts below:
>
> On Thu, Nov 6, 2014 at 7:38 PM, Nirmal Fernando <nirmal070125@gmail.com
> wrote:
>>
>> AFAIU if it is statistics, it's all about random data, samples and
normalization. You don't use all values to do estimations. And this is an
estimation for gradient per say!
>
>
> True, however the random data needs to be accurate as much as possible.
>>>
>>>
>> Well, statistics we are calculating is for a cluster as a whole not
member wise. Since, we autoscale a cluster.
>
>
> Yes for autoscaling a cluster the aggregated statistics should be
calculated against the cluster. However I do not think that we can mix each
statistic accorss members when calculating differences. Different members
of a cluster might be running at different resource usage levels at a given
point of time. Therefore aggregation might needed to be done at the member
level first and then on the cluster level. WDYT?
+1
We need to sample stats and aggregate them on fixed intervals before
calculating the gradients.
Thanks.
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
Sorry for the delayed response Imesh.
On Thu, Nov 6, 2014 at 7:29 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Thanks for your response Nirmal, please see my thoughts below:
>
> On Thu, Nov 6, 2014 at 7:38 PM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
>> AFAIU if it is statistics, it's all about random data, samples and
>> normalization. You don't use all values to do estimations. And this is an
>> estimation for gradient per say!
>>
>
> True, however the random data needs to be accurate as much as possible.
>
Yes, Imesh but this was the first step. We were crawling and let's stand
gradually. :-)
>
>>> Well, statistics we are calculating is for a cluster as a whole not
>> member wise. Since, we autoscale a cluster.
>>
>
> Yes for autoscaling a cluster the aggregated statistics should be
> calculated against the cluster. However I do not think that we can mix each
> statistic accorss members when calculating differences. Different members
> of a cluster might be running at different resource usage levels at a given
> point of time.
>
In my understanding, members of a cluster are homogeneous from the
allocated resource point of view. It's true that their usage levels are
different at a given point of time (which is obvious).
I doubt whether your suggestion is scalable in a system where we have 10s
of clusters and 100s of members in each cluster (since we need to run an
execution plan for each member ). This would be very costly IMO.
Therefore aggregation might needed to be done at the member level first and
> then on the cluster level. WDYT?
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Imesh Gunaratne <im...@apache.org>.
Thanks for your response Nirmal, please see my thoughts below:
On Thu, Nov 6, 2014 at 7:38 PM, Nirmal Fernando <ni...@gmail.com>
wrote:
> AFAIU if it is statistics, it's all about random data, samples and
> normalization. You don't use all values to do estimations. And this is an
> estimation for gradient per say!
>
True, however the random data needs to be accurate as much as possible.
>
>> Well, statistics we are calculating is for a cluster as a whole not
> member wise. Since, we autoscale a cluster.
>
Yes for autoscaling a cluster the aggregated statistics should be
calculated against the cluster. However I do not think that we can mix each
statistic accorss members when calculating differences. Different members
of a cluster might be running at different resource usage levels at a given
point of time. Therefore aggregation might needed to be done at the member
level first and then on the cluster level. WDYT?
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
AFAIU if it is statistics, it's all about random data, samples and
normalization. You don't use all values to do estimations. And this is an
estimation for gradient per say!
On Thu, Nov 6, 2014 at 2:21 PM, Imesh Gunaratne <im...@apache.org> wrote:
> In addition to this as Udara has pointed out, we will need to calculate
> the gradient and the second derivative values for the whole data set
> (rather than using the first and the last values). Otherwise the
> autoscaling decisions we take based on these statistics might not be
> accurate.
>
> On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Raj: Thanks for the clarification, it was my misunderstanding.
>>
>> I did further analysis on this issue, seems like the root cause of this
>> problem resides somewhere else.
>>
>> Currently we have set the cartridge agent's statistics publishing
>> interval to 15 seconds. If so how do we receive series of events within a
>> second?
>>
>
Well, statistics we are calculating is for a cluster as a whole not member
wise. Since, we autoscale a cluster.
> Seems like we are considering statistics events coming from different
>> members for calculating the gradient. This might not be correct, if we were
>> to draw a curve of a data set and consider the gradient of those values,
>> those values may have come from the same source. Otherwise the gradient
>> value we are calculating might not be correct.
>>
>> Thanks
>>
>> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Imagine we put the values in a graph (value against time). The gradient
>>> is the slope of the line. When we draw the line via two very near dots, we
>>> get a high slope/gradient even for a low increase of the values. This
>>> happens when we have only events which are occurred in very near time
>>> intervals.
>>>
>>> On the other hands, when events are distributed among long time period
>>> (are more counts in sample) we don't get this high gradient since the line
>>> goes via higher dots count where we get a reasonable value for gradient.
>>>
>>> As I understood the scenario, it is reasonable to make time gap equals
>>> to millisInSeconds in case time gap is very small.
>>>
>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>> I'd rather keep it simple Akila (this would complex the scenario of
>>> second derivative too) :-) Isn't it?
>>>
>>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>>> ravihansa@wso2.com> wrote:
>>>
>>>> Until t2 - t1 > 1
>>>>
>>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <
>>>> nirmal070125@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>>> ravihansa@wso2.com> wrote:
>>>>>
>>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>>
>>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>>
>>>>>
>>>>> What would be the sufficient value? Theoretically in gradient case
>>>>> it's 2, in second derivative case it's 3. And we're doing that already.
>>>>>
>>>>>>
>>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>>> samples.
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>>> taking the difference*.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Raj: I agree, in this scenario it makes no difference, however
>>>>>>>>> when we are applying a fomular the unit conversion should happen prior to
>>>>>>>>> applying values.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>>>>> division operation to make each operands to seconds, then since both are
>>>>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Still I see a problem with this correction, take the same example:
>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>
>>>>>>>>> *Manual calulation:*
>>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>>>>> 0.005 sec
>>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>>
>>>>>>>>> *According to the current fix:*
>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>> millisecondsForASecond;
>>>>>>>>> double gradient = 0.0;
>>>>>>>>> if (tGap > 0) {
>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>>
>>>>>>>>
>>>>>>>> IMO, this is a sufficient solution.
>>>>>>>>
>>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>>
>>>>>>>> We are taking only first and last values in a window for gradient
>>>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>>
>>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>>> deal with samples which are not good enough.
>>>>>>>>
>>>>>>>> By previous formula, we get -1000. This happened because the sample
>>>>>>>> is not good enough (may be there are few events). Fix makes it to -5.
>>>>>>>>
>>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <gayang@wso2.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>>>>
>>>>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>>
>>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You got it! thanks.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Gayan
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>>> correct here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As I see we are taking the difference of two timestamp values
>>>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>>
>>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>>
>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>>
>>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but
>>>>>>>>>>>> the result's unit will be different.
>>>>>>>>>>>>
>>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>>
>>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>>
>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2
>>>>>>>>>>>>>> - t1)*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug
>>>>>>>>>>>>>> logs;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157 hash:
>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, as you can see the reason behind a large value is when
>>>>>>>>>>>>>> the time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>>> *____
>>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>>> *|
>>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>>> *|____
>>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge
>>>>>>>>>>>>>>> agent also right?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1]
>>>>>>>>>>>>>>>>>> INFO EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too.
>>>>>>>>>>>>>>>>> This is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current
>>>>>>>>>>>>>>>>>> execution plans it took networkPartitionId for the calculation. IMO we need
>>>>>>>>>>>>>>>>>> to update the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in
>>>>>>>>>>>>>>>>>>>> the previous mail.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There
>>>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and
>>>>>>>>>>>>>>>>>>>>>> see what are the load average values sent from the agent. If those values
>>>>>>>>>>>>>>>>>>>>>> are normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar
>>>>>>>>>>>>>>>>>>>>>>> Rajaratnam <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent.
>>>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>
>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Raj
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>> Technical Lead
>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>> Nirmal
>>>>>>>>>>
>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>
>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Imesh Gunaratne
>>>>>>>>>
>>>>>>>>> Technical Lead, WSO2
>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Raj
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Raj
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Akila Ravihansa Perera
>>>>>> Software Engineer, WSO2
>>>>>>
>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nirmal
>>>>>
>>>>> Nirmal Fernando.
>>>>> PPMC Member & Committer of Apache Stratos,
>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Akila Ravihansa Perera
>>>> Software Engineer, WSO2
>>>>
>>>> Blog: http://ravihansa3000.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Imesh Gunaratne <im...@apache.org>.
In addition to this as Udara has pointed out, we will need to calculate the
gradient and the second derivative values for the whole data set (rather
than using the first and the last values). Otherwise the autoscaling
decisions we take based on these statistics might not be accurate.
On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Raj: Thanks for the clarification, it was my misunderstanding.
>
> I did further analysis on this issue, seems like the root cause of this
> problem resides somewhere else.
>
> Currently we have set the cartridge agent's statistics publishing interval
> to 15 seconds. If so how do we receive series of events within a second?
> Seems like we are considering statistics events coming from different
> members for calculating the gradient. This might not be correct, if we were
> to draw a curve of a data set and consider the gradient of those values,
> those values may have come from the same source. Otherwise the gradient
> value we are calculating might not be correct.
>
> Thanks
>
> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi,
>>
>> Imagine we put the values in a graph (value against time). The gradient
>> is the slope of the line. When we draw the line via two very near dots, we
>> get a high slope/gradient even for a low increase of the values. This
>> happens when we have only events which are occurred in very near time
>> intervals.
>>
>> On the other hands, when events are distributed among long time period
>> (are more counts in sample) we don't get this high gradient since the line
>> goes via higher dots count where we get a reasonable value for gradient.
>>
>> As I understood the scenario, it is reasonable to make time gap equals to
>> millisInSeconds in case time gap is very small.
>>
>> Touched, not typed. Erroneous words are a feature, not a typo.
>> I'd rather keep it simple Akila (this would complex the scenario of
>> second derivative too) :-) Isn't it?
>>
>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>> ravihansa@wso2.com> wrote:
>>
>>> Until t2 - t1 > 1
>>>
>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <nirmal070125@gmail.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>> ravihansa@wso2.com> wrote:
>>>>
>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>
>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>
>>>>
>>>> What would be the sufficient value? Theoretically in gradient case it's
>>>> 2, in second derivative case it's 3. And we're doing that already.
>>>>
>>>>>
>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>> samples.
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>> taking the difference*.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Raj: I agree, in this scenario it makes no difference, however when
>>>>>>>> we are applying a fomular the unit conversion should happen prior to
>>>>>>>> applying values.
>>>>>>>>
>>>>>>>
>>>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>>>> division operation to make each operands to seconds, then since both are
>>>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>
>>>>>>>
>>>>>>>> Still I see a problem with this correction, take the same example:
>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>
>>>>>>>> *Manual calulation:*
>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>>>> 0.005 sec
>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>
>>>>>>>> *According to the current fix:*
>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>> millisecondsForASecond;
>>>>>>>> double gradient = 0.0;
>>>>>>>> if (tGap > 0) {
>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>> }
>>>>>>>>
>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>
>>>>>>>
>>>>>>> IMO, this is a sufficient solution.
>>>>>>>
>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>
>>>>>>> We are taking only first and last values in a window for gradient
>>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>
>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>> deal with samples which are not good enough.
>>>>>>>
>>>>>>> By previous formula, we get -1000. This happened because the sample
>>>>>>> is not good enough (may be there are few events). Fix makes it to -5.
>>>>>>>
>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>>
>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>>>
>>>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>
>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You got it! thanks.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Gayan
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>
>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>> correct here.
>>>>>>>>>>>>
>>>>>>>>>>>> As I see we are taking the difference of two timestamp values
>>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>
>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>
>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>
>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>
>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>
>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>
>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>
>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>
>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>>>>>>>> result's unit will be different.
>>>>>>>>>>>
>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>
>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>
>>>>>>>>>>> wdyt?
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>
>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>
>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2
>>>>>>>>>>>>> - t1)*
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>
>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157 hash:
>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>>>
>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, as you can see the reason behind a large value is when the
>>>>>>>>>>>>> time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>> *____
>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>> *|
>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>> *|____
>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent
>>>>>>>>>>>>>> also right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too. This
>>>>>>>>>>>>>>>> is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current execution
>>>>>>>>>>>>>>>>> plans it took networkPartitionId for the calculation. IMO we need to update
>>>>>>>>>>>>>>>>> the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in
>>>>>>>>>>>>>>>>>>> the previous mail.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There
>>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see
>>>>>>>>>>>>>>>>>>>>> what are the load average values sent from the agent. If those values are
>>>>>>>>>>>>>>>>>>>>> normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>> <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent.
>>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>
>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Raj
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>> Technical Lead
>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Nirmal
>>>>>>>>>
>>>>>>>>> Nirmal Fernando.
>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>
>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Imesh Gunaratne
>>>>>>>>
>>>>>>>> Technical Lead, WSO2
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Raj
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Raj
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Akila Ravihansa Perera
>>>>> Software Engineer, WSO2
>>>>>
>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Nirmal
>>>>
>>>> Nirmal Fernando.
>>>> PPMC Member & Committer of Apache Stratos,
>>>> Senior Software Engineer, WSO2 Inc.
>>>>
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Akila Ravihansa Perera
>>> Software Engineer, WSO2
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
On Thu, Nov 6, 2014 at 7:15 PM, Lahiru Sandaruwan <la...@wso2.com> wrote:
> Hi all,
>
> Very interesting discussion and thanks Nirmal for initiating this!
>
> I also did some research on this.
> I think best approach to find gradient and second derivative is to "curve
> fitting". Refer "Locally weighted linear regression" section at [1]. We
> should be calculating gradient and second derivative by differentiating the
> equation of fitted line.
>
> We would need a second degree polynomial fitter for this, where we can use
> Apache commons math library for this. Refer the sample at [2], we need to
> run this with degree 2 instead of 3.
>
> E.g.
> So if get degree 2 polynomial fitter, we will have an equation like below
> where value(v) is our statistic value, time(t) is the time of event,
> gradient(g), and second derivative(sd).
>
> Equation we get from received events,
> v = a*t*t + b*t + c
>
> g = 2*a*t
> sd = 2*a
>
> We can get *g* by applying the median of the time stamp sample.
>
> I have run a sample successfully. I can complete the work on CEP extension
> if you guys agree.
>
Please give us some time to go through it Lahiru :) may be this weekend.
Thanks.
> In addition to this, we should also restrict the sample size as Akila
> suggested. Ideally we should receive 6 events per minute for memberwise
> stats and "6 * no. of instances" for clusterwise stats. We should put a
> constraint with this much of percentage(say 80%) should be there to
> calculate *g *and* sd.*
>
> Also we should not calculated gradient of stats sent from different
> members in clusterwise scenario as Imesh mentioned. That causes this type
> of small time gap issues.
>
> Thanks.
>
> [1] http://cs229.stanford.edu/notes/cs229-notes1.pdf
> [2] http://commons.apache.org/proper/commons-math/userguide/fitting.html
>
> On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Raj: Thanks for the clarification, it was my misunderstanding.
>>
>> I did further analysis on this issue, seems like the root cause of this
>> problem resides somewhere else.
>>
>> Currently we have set the cartridge agent's statistics publishing
>> interval to 15 seconds. If so how do we receive series of events within a
>> second? Seems like we are considering statistics events coming from
>> different members for calculating the gradient. This might not be correct,
>> if we were to draw a curve of a data set and consider the gradient of those
>> values, those values may have come from the same source. Otherwise the
>> gradient value we are calculating might not be correct.
>>
>> Thanks
>>
>> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Imagine we put the values in a graph (value against time). The gradient
>>> is the slope of the line. When we draw the line via two very near dots, we
>>> get a high slope/gradient even for a low increase of the values. This
>>> happens when we have only events which are occurred in very near time
>>> intervals.
>>>
>>> On the other hands, when events are distributed among long time period
>>> (are more counts in sample) we don't get this high gradient since the line
>>> goes via higher dots count where we get a reasonable value for gradient.
>>>
>>> As I understood the scenario, it is reasonable to make time gap equals
>>> to millisInSeconds in case time gap is very small.
>>>
>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>> I'd rather keep it simple Akila (this would complex the scenario of
>>> second derivative too) :-) Isn't it?
>>>
>>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>>> ravihansa@wso2.com> wrote:
>>>
>>>> Until t2 - t1 > 1
>>>>
>>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <
>>>> nirmal070125@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>>> ravihansa@wso2.com> wrote:
>>>>>
>>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>>
>>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>>
>>>>>
>>>>> What would be the sufficient value? Theoretically in gradient case
>>>>> it's 2, in second derivative case it's 3. And we're doing that already.
>>>>>
>>>>>>
>>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>>> samples.
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>>> taking the difference*.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Raj: I agree, in this scenario it makes no difference, however
>>>>>>>>> when we are applying a fomular the unit conversion should happen prior to
>>>>>>>>> applying values.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>>>>> division operation to make each operands to seconds, then since both are
>>>>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Still I see a problem with this correction, take the same example:
>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>
>>>>>>>>> *Manual calulation:*
>>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>>>>> 0.005 sec
>>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>>
>>>>>>>>> *According to the current fix:*
>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>> millisecondsForASecond;
>>>>>>>>> double gradient = 0.0;
>>>>>>>>> if (tGap > 0) {
>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>>
>>>>>>>>
>>>>>>>> IMO, this is a sufficient solution.
>>>>>>>>
>>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>>
>>>>>>>> We are taking only first and last values in a window for gradient
>>>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>>
>>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>>> deal with samples which are not good enough.
>>>>>>>>
>>>>>>>> By previous formula, we get -1000. This happened because the sample
>>>>>>>> is not good enough (may be there are few events). Fix makes it to -5.
>>>>>>>>
>>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <gayang@wso2.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>>>>
>>>>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>>
>>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You got it! thanks.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Gayan
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>>> correct here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As I see we are taking the difference of two timestamp values
>>>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>>
>>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>>
>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>>
>>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but
>>>>>>>>>>>> the result's unit will be different.
>>>>>>>>>>>>
>>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>>
>>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>>
>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2
>>>>>>>>>>>>>> - t1)*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug
>>>>>>>>>>>>>> logs;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157 hash:
>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, as you can see the reason behind a large value is when
>>>>>>>>>>>>>> the time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>>> *____
>>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>>> *|
>>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>>> *|____
>>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge
>>>>>>>>>>>>>>> agent also right?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1]
>>>>>>>>>>>>>>>>>> INFO EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too.
>>>>>>>>>>>>>>>>> This is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current
>>>>>>>>>>>>>>>>>> execution plans it took networkPartitionId for the calculation. IMO we need
>>>>>>>>>>>>>>>>>> to update the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in
>>>>>>>>>>>>>>>>>>>> the previous mail.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There
>>>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and
>>>>>>>>>>>>>>>>>>>>>> see what are the load average values sent from the agent. If those values
>>>>>>>>>>>>>>>>>>>>>> are normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar
>>>>>>>>>>>>>>>>>>>>>>> Rajaratnam <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent.
>>>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>
>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Raj
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>> Technical Lead
>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>> Nirmal
>>>>>>>>>>
>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>
>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Imesh Gunaratne
>>>>>>>>>
>>>>>>>>> Technical Lead, WSO2
>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Raj
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Raj
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Akila Ravihansa Perera
>>>>>> Software Engineer, WSO2
>>>>>>
>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nirmal
>>>>>
>>>>> Nirmal Fernando.
>>>>> PPMC Member & Committer of Apache Stratos,
>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Akila Ravihansa Perera
>>>> Software Engineer, WSO2
>>>>
>>>> Blog: http://ravihansa3000.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
--
Raj
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Lahiru Sandaruwan <la...@wso2.com>.
On Tue, Nov 11, 2014 at 3:06 PM, Lakmal Warusawithana <la...@wso2.com>
wrote:
>
>
> On Sat, Nov 8, 2014 at 9:42 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
>>
>>
>> On Sat, Nov 8, 2014 at 12:49 AM, Nirmal Fernando <ni...@gmail.com>
>> wrote:
>>
>>> Hi Lahiru,
>>>
>>> All these are improvements and should have discussed in a new thread
>>> IMO. (these are buried now)
>>>
>>
>> +1. I will send a complete proposal to a new thread.
>>
>>
> Do we have new thread?
>
WIll start soon.
Thanks.
>
> IMO, we should go for the following fix, otherwise we can't defend on the
> logic behind this.
>
> - CEP should calculate first, second derivatives against individual
> member, not the network cluster.
> - AutoScaler should calculate prediction values for individual member.
> Then AutoScaler should calculate the average of prediction values against
> network partition wise.
> - Based on this, we should do the scale up or down.
>
>
> @Nirmal, yes this will hit some load specially on CEP, but my POV this is
> the correct solution, and we should go for that.
>
>
>> Thanks.
>>
>>>
>>> What we currently have is linear gradient, which is the simplest form
>>> (first step!)
>>>
>>> On Thu, Nov 6, 2014 at 2:45 PM, Lahiru Sandaruwan <la...@wso2.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Very interesting discussion and thanks Nirmal for initiating this!
>>>>
>>>> I also did some research on this.
>>>> I think best approach to find gradient and second derivative is to
>>>> "curve fitting". Refer "Locally weighted linear regression" section at [1].
>>>> We should be calculating gradient and second derivative by differentiating
>>>> the equation of fitted line.
>>>>
>>>> We would need a second degree polynomial fitter for this, where we can
>>>> use Apache commons math library for this. Refer the sample at [2], we need
>>>> to run this with degree 2 instead of 3.
>>>>
>>>> E.g.
>>>> So if get degree 2 polynomial fitter, we will have an equation like
>>>> below where value(v) is our statistic value, time(t) is the time of event,
>>>> gradient(g), and second derivative(sd).
>>>>
>>>> Equation we get from received events,
>>>> v = a*t*t + b*t + c
>>>>
>>>> g = 2*a*t
>>>>
>>>
>>> g= 2*a*t + b
>>>
>>>
>>>> sd = 2*a
>>>>
>>>> We can get *g* by applying the median of the time stamp sample.
>>>>
>>>
>>> If you use median of the timestamp sample, g would grow with time?!
>>>
>>>
>>>>
>>>> I have run a sample successfully. I can complete the work on CEP
>>>> extension if you guys agree.
>>>>
>>>> In addition to this, we should also restrict the sample size as Akila
>>>> suggested. Ideally we should receive 6 events per minute for memberwise
>>>> stats and "6 * no. of instances" for clusterwise stats.
>>>>
>>>
>>> From where did you get '6' ?
>>>
>>>
>>>> We should put a constraint with this much of percentage(say 80%) should
>>>> be there to calculate *g *and* sd.*
>>>>
>>>
>>> This much of a percentage from what value?
>>>
>>>>
>>>> Also we should not calculated gradient of stats sent from different
>>>> members in clusterwise scenario as Imesh mentioned. That causes this type
>>>> of small time gap issues.
>>>>
>>>> Thanks.
>>>>
>>>> [1] http://cs229.stanford.edu/notes/cs229-notes1.pdf
>>>> [2]
>>>> http://commons.apache.org/proper/commons-math/userguide/fitting.html
>>>>
>>>> On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Raj: Thanks for the clarification, it was my misunderstanding.
>>>>>
>>>>> I did further analysis on this issue, seems like the root cause of
>>>>> this problem resides somewhere else.
>>>>>
>>>>> Currently we have set the cartridge agent's statistics publishing
>>>>> interval to 15 seconds. If so how do we receive series of events within a
>>>>> second? Seems like we are considering statistics events coming from
>>>>> different members for calculating the gradient. This might not be correct,
>>>>> if we were to draw a curve of a data set and consider the gradient of those
>>>>> values, those values may have come from the same source. Otherwise the
>>>>> gradient value we are calculating might not be correct.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Imagine we put the values in a graph (value against time). The
>>>>>> gradient is the slope of the line. When we draw the line via two very near
>>>>>> dots, we get a high slope/gradient even for a low increase of the values.
>>>>>> This happens when we have only events which are occurred in very near time
>>>>>> intervals.
>>>>>>
>>>>>> On the other hands, when events are distributed among long time
>>>>>> period (are more counts in sample) we don't get this high gradient since
>>>>>> the line goes via higher dots count where we get a reasonable value for
>>>>>> gradient.
>>>>>>
>>>>>> As I understood the scenario, it is reasonable to make time gap
>>>>>> equals to millisInSeconds in case time gap is very small.
>>>>>>
>>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>>> I'd rather keep it simple Akila (this would complex the scenario of
>>>>>> second derivative too) :-) Isn't it?
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>>>>>> ravihansa@wso2.com> wrote:
>>>>>>
>>>>>>> Until t2 - t1 > 1
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <
>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>>>>>> ravihansa@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>>>>>
>>>>>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>>>>>
>>>>>>>>
>>>>>>>> What would be the sufficient value? Theoretically in gradient case
>>>>>>>> it's 2, in second derivative case it's 3. And we're doing that already.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>>>>>> samples.
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <
>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Raj: I agree, in this scenario it makes no difference, however
>>>>>>>>>>>> when we are applying a fomular the unit conversion should happen prior to
>>>>>>>>>>>> applying values.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Not exactly Imesh. We can do operations if operands are in the
>>>>>>>>>>> same units. So we can do subtract operation in milliseconds. And then we
>>>>>>>>>>> can do division operation to make it in seconds. On the other hand, we can
>>>>>>>>>>> do a division operation to make each operands to seconds, then since both
>>>>>>>>>>> are seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Still I see a problem with this correction, take the same
>>>>>>>>>>>> example:
>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>
>>>>>>>>>>>> *Manual calulation:*
>>>>>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000
>>>>>>>>>>>> = 0.005 sec
>>>>>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>>>>>
>>>>>>>>>>>> *According to the current fix:*
>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> IMO, this is a sufficient solution.
>>>>>>>>>>>
>>>>>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>>>>>
>>>>>>>>>>> We are taking only first and last values in a window for
>>>>>>>>>>> gradient calculation. If there are more events in that window, and if that
>>>>>>>>>>> is a good sample, tGap will be taking some higher values than 1s. This is
>>>>>>>>>>> usual situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>>>>>
>>>>>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>>>>>> deal with samples which are not good enough.
>>>>>>>>>>>
>>>>>>>>>>> By previous formula, we get -1000. This happened because the
>>>>>>>>>>> sample is not good enough (may be there are few events). Fix makes it to
>>>>>>>>>>> -5.
>>>>>>>>>>>
>>>>>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <
>>>>>>>>>>>>> gayang@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> AFAIU this is not related with the gradient calculation
>>>>>>>>>>>>>> formula that we used. But It is related with the how we calculate gradient
>>>>>>>>>>>>>> when the denominator become a small number.IMO when the divisor
>>>>>>>>>>>>>> (timeInSeconds) approaches to the small number we do need to normalize the
>>>>>>>>>>>>>> result.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IMO we do have max and min scale up and down values. So based
>>>>>>>>>>>>>> on that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> You got it! thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 -
>>>>>>>>>>>>>> t1 : millisecondsForASecond;
>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Gayan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>>>>>> correct here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As I see we are taking the difference of two timestamp
>>>>>>>>>>>>>>>> values and then dividing it by 1000. The corect way might be to first
>>>>>>>>>>>>>>>> divide each value by 1000 and then take the difference.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by
>>>>>>>>>>>>>>> 1000 and taking the difference*.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On the other hand we might not need to convert these values
>>>>>>>>>>>>>>>> to seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001
>>>>>>>>>>>>>>>> First val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 -
>>>>>>>>>>>>>>>> t1 : millisecondsForASecond;
>>>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but
>>>>>>>>>>>>>>> the result's unit will be different.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) /
>>>>>>>>>>>>>>>>> (t2 - t1)*
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug
>>>>>>>>>>>>>>>>> logs;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>>>>> hash: 155426542
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610
>>>>>>>>>>>>>>>>> t2: 1415213232154 hash: 155426542
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So, as you can see the reason behind a large value is when
>>>>>>>>>>>>>>>>> the time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>>>>>> *____
>>>>>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>>>>>> *|
>>>>>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>>>>>> *|____
>>>>>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we
>>>>>>>>>>>>>>>>>>> can compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge
>>>>>>>>>>>>>>>>>> agent also right?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I
>>>>>>>>>>>>>>>>>>>>> go though the logs I'm unable to find the high input values for load
>>>>>>>>>>>>>>>>>>>>> average. But output have high values.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1]
>>>>>>>>>>>>>>>>>>>>> INFO EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too.
>>>>>>>>>>>>>>>>>>>> This is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current
>>>>>>>>>>>>>>>>>>>>> execution plans it took networkPartitionId for the calculation. IMO we need
>>>>>>>>>>>>>>>>>>>>> to update the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>> <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached
>>>>>>>>>>>>>>>>>>>>>>> in the previous mail.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>> <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent.
>>>>>>>>>>>>>>>>>>>>>>>> There are traces for CEP sending large values. Find the cep trace log here
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and
>>>>>>>>>>>>>>>>>>>>>>>>> see what are the load average values sent from the agent. If those values
>>>>>>>>>>>>>>>>>>>>>>>>> are normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis
>>>>>>>>>>>>>>>>>>>>>>>>> <ch...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same
>>>>>>>>>>>>>>>>>>>>>>>>>> time period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar
>>>>>>>>>>>>>>>>>>>>>>>>>> Rajaratnam <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python
>>>>>>>>>>>>>>>>>>>>>>>>>>> agent. And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> [value] 200000.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>>>>> 0.11670045
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of load avg event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete
>>>>>>>>>>>>>>>>>>>>>>>>>>> containers rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check]
>>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Minimum check executed for :
>>>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting
>>>>>>>>>>>>>>>>>>>>>>>>>>> the value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 ,
>>>>>>>>>>>>>>>>>>>>>>>>>>> [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting
>>>>>>>>>>>>>>>>>>>>>>>>>>> the value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 ,
>>>>>>>>>>>>>>>>>>>>>>>>>>> [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting
>>>>>>>>>>>>>>>>>>>>>>>>>>> the value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up]
>>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720]
>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO
>>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>>>>>>>>>>>>>>>>>> - CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758]
>>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Cloud Controller is delegating request to update a replication
>>>>>>>>>>>>>>>>>>>>>>>>>>> controller php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Raj
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>
>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Raj
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Raj
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Akila Ravihansa Perera
>>>>>>>>> Software Engineer, WSO2
>>>>>>>>>
>>>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Nirmal
>>>>>>>>
>>>>>>>> Nirmal Fernando.
>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>
>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Akila Ravihansa Perera
>>>>>>> Software Engineer, WSO2
>>>>>>>
>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Nirmal
>>>>>>
>>>>>> Nirmal Fernando.
>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>
>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Lahiru Sandaruwan
>>>> Committer and PMC member, Apache Stratos,
>>>> Senior Software Engineer,
>>>> WSO2 Inc., http://wso2.com
>>>> lean.enterprise.middleware
>>>>
>>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>
>
> --
> Lakmal Warusawithana
> Vice President, Apache Stratos
> Director - Cloud Architecture; WSO2 Inc.
> Mobile : +94714289692
> Blog : http://lakmalsview.blogspot.com/
>
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Lakmal Warusawithana <la...@wso2.com>.
On Sat, Nov 8, 2014 at 9:42 AM, Lahiru Sandaruwan <la...@wso2.com> wrote:
>
>
> On Sat, Nov 8, 2014 at 12:49 AM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
>> Hi Lahiru,
>>
>> All these are improvements and should have discussed in a new thread IMO.
>> (these are buried now)
>>
>
> +1. I will send a complete proposal to a new thread.
>
>
Do we have new thread?
IMO, we should go for the following fix, otherwise we can't defend on the
logic behind this.
- CEP should calculate first, second derivatives against individual
member, not the network cluster.
- AutoScaler should calculate prediction values for individual member.
Then AutoScaler should calculate the average of prediction values against
network partition wise.
- Based on this, we should do the scale up or down.
@Nirmal, yes this will hit some load specially on CEP, but my POV this is
the correct solution, and we should go for that.
> Thanks.
>
>>
>> What we currently have is linear gradient, which is the simplest form
>> (first step!)
>>
>> On Thu, Nov 6, 2014 at 2:45 PM, Lahiru Sandaruwan <la...@wso2.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> Very interesting discussion and thanks Nirmal for initiating this!
>>>
>>> I also did some research on this.
>>> I think best approach to find gradient and second derivative is to
>>> "curve fitting". Refer "Locally weighted linear regression" section at [1].
>>> We should be calculating gradient and second derivative by differentiating
>>> the equation of fitted line.
>>>
>>> We would need a second degree polynomial fitter for this, where we can
>>> use Apache commons math library for this. Refer the sample at [2], we need
>>> to run this with degree 2 instead of 3.
>>>
>>> E.g.
>>> So if get degree 2 polynomial fitter, we will have an equation like
>>> below where value(v) is our statistic value, time(t) is the time of event,
>>> gradient(g), and second derivative(sd).
>>>
>>> Equation we get from received events,
>>> v = a*t*t + b*t + c
>>>
>>> g = 2*a*t
>>>
>>
>> g= 2*a*t + b
>>
>>
>>> sd = 2*a
>>>
>>> We can get *g* by applying the median of the time stamp sample.
>>>
>>
>> If you use median of the timestamp sample, g would grow with time?!
>>
>>
>>>
>>> I have run a sample successfully. I can complete the work on CEP
>>> extension if you guys agree.
>>>
>>> In addition to this, we should also restrict the sample size as Akila
>>> suggested. Ideally we should receive 6 events per minute for memberwise
>>> stats and "6 * no. of instances" for clusterwise stats.
>>>
>>
>> From where did you get '6' ?
>>
>>
>>> We should put a constraint with this much of percentage(say 80%) should
>>> be there to calculate *g *and* sd.*
>>>
>>
>> This much of a percentage from what value?
>>
>>>
>>> Also we should not calculated gradient of stats sent from different
>>> members in clusterwise scenario as Imesh mentioned. That causes this type
>>> of small time gap issues.
>>>
>>> Thanks.
>>>
>>> [1] http://cs229.stanford.edu/notes/cs229-notes1.pdf
>>> [2] http://commons.apache.org/proper/commons-math/userguide/fitting.html
>>>
>>> On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> Raj: Thanks for the clarification, it was my misunderstanding.
>>>>
>>>> I did further analysis on this issue, seems like the root cause of this
>>>> problem resides somewhere else.
>>>>
>>>> Currently we have set the cartridge agent's statistics publishing
>>>> interval to 15 seconds. If so how do we receive series of events within a
>>>> second? Seems like we are considering statistics events coming from
>>>> different members for calculating the gradient. This might not be correct,
>>>> if we were to draw a curve of a data set and consider the gradient of those
>>>> values, those values may have come from the same source. Otherwise the
>>>> gradient value we are calculating might not be correct.
>>>>
>>>> Thanks
>>>>
>>>> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Imagine we put the values in a graph (value against time). The
>>>>> gradient is the slope of the line. When we draw the line via two very near
>>>>> dots, we get a high slope/gradient even for a low increase of the values.
>>>>> This happens when we have only events which are occurred in very near time
>>>>> intervals.
>>>>>
>>>>> On the other hands, when events are distributed among long time period
>>>>> (are more counts in sample) we don't get this high gradient since the line
>>>>> goes via higher dots count where we get a reasonable value for gradient.
>>>>>
>>>>> As I understood the scenario, it is reasonable to make time gap equals
>>>>> to millisInSeconds in case time gap is very small.
>>>>>
>>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>>> I'd rather keep it simple Akila (this would complex the scenario of
>>>>> second derivative too) :-) Isn't it?
>>>>>
>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>>>>> ravihansa@wso2.com> wrote:
>>>>>
>>>>>> Until t2 - t1 > 1
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <
>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>>>>> ravihansa@wso2.com> wrote:
>>>>>>>
>>>>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>>>>
>>>>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>>>>
>>>>>>>
>>>>>>> What would be the sufficient value? Theoretically in gradient case
>>>>>>> it's 2, in second derivative case it's 3. And we're doing that already.
>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>>>>> samples.
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <
>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Raj: I agree, in this scenario it makes no difference, however
>>>>>>>>>>> when we are applying a fomular the unit conversion should happen prior to
>>>>>>>>>>> applying values.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Not exactly Imesh. We can do operations if operands are in the
>>>>>>>>>> same units. So we can do subtract operation in milliseconds. And then we
>>>>>>>>>> can do division operation to make it in seconds. On the other hand, we can
>>>>>>>>>> do a division operation to make each operands to seconds, then since both
>>>>>>>>>> are seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Still I see a problem with this correction, take the same
>>>>>>>>>>> example:
>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>
>>>>>>>>>>> *Manual calulation:*
>>>>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000
>>>>>>>>>>> = 0.005 sec
>>>>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>>>>
>>>>>>>>>>> *According to the current fix:*
>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> IMO, this is a sufficient solution.
>>>>>>>>>>
>>>>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>>>>
>>>>>>>>>> We are taking only first and last values in a window for gradient
>>>>>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>>>>
>>>>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>>>>> deal with samples which are not good enough.
>>>>>>>>>>
>>>>>>>>>> By previous formula, we get -1000. This happened because the
>>>>>>>>>> sample is not good enough (may be there are few events). Fix makes it to
>>>>>>>>>> -5.
>>>>>>>>>>
>>>>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <
>>>>>>>>>>>> gayang@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> AFAIU this is not related with the gradient calculation
>>>>>>>>>>>>> formula that we used. But It is related with the how we calculate gradient
>>>>>>>>>>>>> when the denominator become a small number.IMO when the divisor
>>>>>>>>>>>>> (timeInSeconds) approaches to the small number we do need to normalize the
>>>>>>>>>>>>> result.
>>>>>>>>>>>>>
>>>>>>>>>>>>> IMO we do have max and min scale up and down values. So based
>>>>>>>>>>>>> on that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> You got it! thanks.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Gayan
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>>>>> correct here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As I see we are taking the difference of two timestamp
>>>>>>>>>>>>>>> values and then dividing it by 1000. The corect way might be to first
>>>>>>>>>>>>>>> divide each value by 1000 and then take the difference.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On the other hand we might not need to convert these values
>>>>>>>>>>>>>>> to seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001
>>>>>>>>>>>>>>> First val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 -
>>>>>>>>>>>>>>> t1 : millisecondsForASecond;
>>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but
>>>>>>>>>>>>>> the result's unit will be different.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) /
>>>>>>>>>>>>>>>> (t2 - t1)*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug
>>>>>>>>>>>>>>>> logs;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>>>> hash: 155426542
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610
>>>>>>>>>>>>>>>> t2: 1415213232154 hash: 155426542
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, as you can see the reason behind a large value is when
>>>>>>>>>>>>>>>> the time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>>>>> *____
>>>>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>>>>> *|
>>>>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>>>>> *|____
>>>>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we
>>>>>>>>>>>>>>>>>> can compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge
>>>>>>>>>>>>>>>>> agent also right?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1]
>>>>>>>>>>>>>>>>>>>> INFO EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too.
>>>>>>>>>>>>>>>>>>> This is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current
>>>>>>>>>>>>>>>>>>>> execution plans it took networkPartitionId for the calculation. IMO we need
>>>>>>>>>>>>>>>>>>>> to update the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached
>>>>>>>>>>>>>>>>>>>>>> in the previous mail.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>> <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent.
>>>>>>>>>>>>>>>>>>>>>>> There are traces for CEP sending large values. Find the cep trace log here
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and
>>>>>>>>>>>>>>>>>>>>>>>> see what are the load average values sent from the agent. If those values
>>>>>>>>>>>>>>>>>>>>>>>> are normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis
>>>>>>>>>>>>>>>>>>>>>>>> <ch...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same
>>>>>>>>>>>>>>>>>>>>>>>>> time period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar
>>>>>>>>>>>>>>>>>>>>>>>>> Rajaratnam <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python
>>>>>>>>>>>>>>>>>>>>>>>>>> agent. And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> [value] 200000.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>>>> 0.11670045
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of Memory Consumption event: [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Second Derivation of load avg event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load
>>>>>>>>>>>>>>>>>>>>>>>>>> average stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Avg Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check]
>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete
>>>>>>>>>>>>>>>>>>>>>>>>>> containers rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check]
>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Minimum check executed for :
>>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting
>>>>>>>>>>>>>>>>>>>>>>>>>> the value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 ,
>>>>>>>>>>>>>>>>>>>>>>>>>> [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting
>>>>>>>>>>>>>>>>>>>>>>>>>> the value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 ,
>>>>>>>>>>>>>>>>>>>>>>>>>> [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting
>>>>>>>>>>>>>>>>>>>>>>>>>> the value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up]
>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] : php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720]
>>>>>>>>>>>>>>>>>>>>>>>>>> INFO
>>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>>>>>>>>>>>>>>>>> - CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758]
>>>>>>>>>>>>>>>>>>>>>>>>>> DEBUG {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
>>>>>>>>>>>>>>>>>>>>>>>>>> - Cloud Controller is delegating request to update a replication
>>>>>>>>>>>>>>>>>>>>>>>>>> controller php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Raj
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>
>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>
>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>
>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Raj
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Raj
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Akila Ravihansa Perera
>>>>>>>> Software Engineer, WSO2
>>>>>>>>
>>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Nirmal
>>>>>>>
>>>>>>> Nirmal Fernando.
>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>
>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Akila Ravihansa Perera
>>>>>> Software Engineer, WSO2
>>>>>>
>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nirmal
>>>>>
>>>>> Nirmal Fernando.
>>>>> PPMC Member & Committer of Apache Stratos,
>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Lahiru Sandaruwan
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
--
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Lahiru Sandaruwan <la...@wso2.com>.
On Sat, Nov 8, 2014 at 12:49 AM, Nirmal Fernando <ni...@gmail.com>
wrote:
> Hi Lahiru,
>
> All these are improvements and should have discussed in a new thread IMO.
> (these are buried now)
>
+1. I will send a complete proposal to a new thread.
Thanks.
>
> What we currently have is linear gradient, which is the simplest form
> (first step!)
>
> On Thu, Nov 6, 2014 at 2:45 PM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
>> Hi all,
>>
>> Very interesting discussion and thanks Nirmal for initiating this!
>>
>> I also did some research on this.
>> I think best approach to find gradient and second derivative is to "curve
>> fitting". Refer "Locally weighted linear regression" section at [1]. We
>> should be calculating gradient and second derivative by differentiating the
>> equation of fitted line.
>>
>> We would need a second degree polynomial fitter for this, where we can
>> use Apache commons math library for this. Refer the sample at [2], we need
>> to run this with degree 2 instead of 3.
>>
>> E.g.
>> So if get degree 2 polynomial fitter, we will have an equation like below
>> where value(v) is our statistic value, time(t) is the time of event,
>> gradient(g), and second derivative(sd).
>>
>> Equation we get from received events,
>> v = a*t*t + b*t + c
>>
>> g = 2*a*t
>>
>
> g= 2*a*t + b
>
>
>> sd = 2*a
>>
>> We can get *g* by applying the median of the time stamp sample.
>>
>
> If you use median of the timestamp sample, g would grow with time?!
>
>
>>
>> I have run a sample successfully. I can complete the work on CEP
>> extension if you guys agree.
>>
>> In addition to this, we should also restrict the sample size as Akila
>> suggested. Ideally we should receive 6 events per minute for memberwise
>> stats and "6 * no. of instances" for clusterwise stats.
>>
>
> From where did you get '6' ?
>
>
>> We should put a constraint with this much of percentage(say 80%) should
>> be there to calculate *g *and* sd.*
>>
>
> This much of a percentage from what value?
>
>>
>> Also we should not calculated gradient of stats sent from different
>> members in clusterwise scenario as Imesh mentioned. That causes this type
>> of small time gap issues.
>>
>> Thanks.
>>
>> [1] http://cs229.stanford.edu/notes/cs229-notes1.pdf
>> [2] http://commons.apache.org/proper/commons-math/userguide/fitting.html
>>
>> On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org> wrote:
>>
>>> Raj: Thanks for the clarification, it was my misunderstanding.
>>>
>>> I did further analysis on this issue, seems like the root cause of this
>>> problem resides somewhere else.
>>>
>>> Currently we have set the cartridge agent's statistics publishing
>>> interval to 15 seconds. If so how do we receive series of events within a
>>> second? Seems like we are considering statistics events coming from
>>> different members for calculating the gradient. This might not be correct,
>>> if we were to draw a curve of a data set and consider the gradient of those
>>> values, those values may have come from the same source. Otherwise the
>>> gradient value we are calculating might not be correct.
>>>
>>> Thanks
>>>
>>> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Imagine we put the values in a graph (value against time). The gradient
>>>> is the slope of the line. When we draw the line via two very near dots, we
>>>> get a high slope/gradient even for a low increase of the values. This
>>>> happens when we have only events which are occurred in very near time
>>>> intervals.
>>>>
>>>> On the other hands, when events are distributed among long time period
>>>> (are more counts in sample) we don't get this high gradient since the line
>>>> goes via higher dots count where we get a reasonable value for gradient.
>>>>
>>>> As I understood the scenario, it is reasonable to make time gap equals
>>>> to millisInSeconds in case time gap is very small.
>>>>
>>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>>> I'd rather keep it simple Akila (this would complex the scenario of
>>>> second derivative too) :-) Isn't it?
>>>>
>>>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>>>> ravihansa@wso2.com> wrote:
>>>>
>>>>> Until t2 - t1 > 1
>>>>>
>>>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <
>>>>> nirmal070125@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>>>> ravihansa@wso2.com> wrote:
>>>>>>
>>>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>>>
>>>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>>>
>>>>>>
>>>>>> What would be the sufficient value? Theoretically in gradient case
>>>>>> it's 2, in second derivative case it's 3. And we're doing that already.
>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>>>> samples.
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <imesh@apache.org
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Raj: I agree, in this scenario it makes no difference, however
>>>>>>>>>> when we are applying a fomular the unit conversion should happen prior to
>>>>>>>>>> applying values.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Not exactly Imesh. We can do operations if operands are in the
>>>>>>>>> same units. So we can do subtract operation in milliseconds. And then we
>>>>>>>>> can do division operation to make it in seconds. On the other hand, we can
>>>>>>>>> do a division operation to make each operands to seconds, then since both
>>>>>>>>> are seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Still I see a problem with this correction, take the same example:
>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>
>>>>>>>>>> *Manual calulation:*
>>>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>>>>>> 0.005 sec
>>>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>>>
>>>>>>>>>> *According to the current fix:*
>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> IMO, this is a sufficient solution.
>>>>>>>>>
>>>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>>>
>>>>>>>>> We are taking only first and last values in a window for gradient
>>>>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>>>
>>>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>>>> deal with samples which are not good enough.
>>>>>>>>>
>>>>>>>>> By previous formula, we get -1000. This happened because the
>>>>>>>>> sample is not good enough (may be there are few events). Fix makes it to
>>>>>>>>> -5.
>>>>>>>>>
>>>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <
>>>>>>>>>>> gayang@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>>>>>
>>>>>>>>>>>> IMO we do have max and min scale up and down values. So based
>>>>>>>>>>>> on that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>>>
>>>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> You got it! thanks.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Gayan
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>>>> correct here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As I see we are taking the difference of two timestamp values
>>>>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On the other hand we might not need to convert these values
>>>>>>>>>>>>>> to seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001
>>>>>>>>>>>>>> First val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 -
>>>>>>>>>>>>>> t1 : millisecondsForASecond;
>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but
>>>>>>>>>>>>> the result's unit will be different.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>>>
>>>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) /
>>>>>>>>>>>>>>> (t2 - t1)*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug
>>>>>>>>>>>>>>> logs;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>>> hash: 155426542
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, as you can see the reason behind a large value is when
>>>>>>>>>>>>>>> the time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>>>> *____
>>>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>>>> *|
>>>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>>>> *|____
>>>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we
>>>>>>>>>>>>>>>>> can compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge
>>>>>>>>>>>>>>>> agent also right?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1]
>>>>>>>>>>>>>>>>>>> INFO EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too.
>>>>>>>>>>>>>>>>>> This is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current
>>>>>>>>>>>>>>>>>>> execution plans it took networkPartitionId for the calculation. IMO we need
>>>>>>>>>>>>>>>>>>> to update the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in
>>>>>>>>>>>>>>>>>>>>> the previous mail.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There
>>>>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and
>>>>>>>>>>>>>>>>>>>>>>> see what are the load average values sent from the agent. If those values
>>>>>>>>>>>>>>>>>>>>>>> are normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same
>>>>>>>>>>>>>>>>>>>>>>>> time period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar
>>>>>>>>>>>>>>>>>>>>>>>> Rajaratnam <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent.
>>>>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Raj
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Nirmal
>>>>>>>>>>>
>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>
>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>
>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Raj
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Raj
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Akila Ravihansa Perera
>>>>>>> Software Engineer, WSO2
>>>>>>>
>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Nirmal
>>>>>>
>>>>>> Nirmal Fernando.
>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>
>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Akila Ravihansa Perera
>>>>> Software Engineer, WSO2
>>>>>
>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Nirmal
>>>>
>>>> Nirmal Fernando.
>>>> PPMC Member & Committer of Apache Stratos,
>>>> Senior Software Engineer, WSO2 Inc.
>>>>
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
Hi Lahiru,
All these are improvements and should have discussed in a new thread IMO.
(these are buried now)
What we currently have is linear gradient, which is the simplest form
(first step!)
On Thu, Nov 6, 2014 at 2:45 PM, Lahiru Sandaruwan <la...@wso2.com> wrote:
> Hi all,
>
> Very interesting discussion and thanks Nirmal for initiating this!
>
> I also did some research on this.
> I think best approach to find gradient and second derivative is to "curve
> fitting". Refer "Locally weighted linear regression" section at [1]. We
> should be calculating gradient and second derivative by differentiating the
> equation of fitted line.
>
> We would need a second degree polynomial fitter for this, where we can use
> Apache commons math library for this. Refer the sample at [2], we need to
> run this with degree 2 instead of 3.
>
> E.g.
> So if get degree 2 polynomial fitter, we will have an equation like below
> where value(v) is our statistic value, time(t) is the time of event,
> gradient(g), and second derivative(sd).
>
> Equation we get from received events,
> v = a*t*t + b*t + c
>
> g = 2*a*t
>
g= 2*a*t + b
> sd = 2*a
>
> We can get *g* by applying the median of the time stamp sample.
>
If you use median of the timestamp sample, g would grow with time?!
>
> I have run a sample successfully. I can complete the work on CEP extension
> if you guys agree.
>
> In addition to this, we should also restrict the sample size as Akila
> suggested. Ideally we should receive 6 events per minute for memberwise
> stats and "6 * no. of instances" for clusterwise stats.
>
>From where did you get '6' ?
> We should put a constraint with this much of percentage(say 80%) should be
> there to calculate *g *and* sd.*
>
This much of a percentage from what value?
>
> Also we should not calculated gradient of stats sent from different
> members in clusterwise scenario as Imesh mentioned. That causes this type
> of small time gap issues.
>
> Thanks.
>
> [1] http://cs229.stanford.edu/notes/cs229-notes1.pdf
> [2] http://commons.apache.org/proper/commons-math/userguide/fitting.html
>
> On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Raj: Thanks for the clarification, it was my misunderstanding.
>>
>> I did further analysis on this issue, seems like the root cause of this
>> problem resides somewhere else.
>>
>> Currently we have set the cartridge agent's statistics publishing
>> interval to 15 seconds. If so how do we receive series of events within a
>> second? Seems like we are considering statistics events coming from
>> different members for calculating the gradient. This might not be correct,
>> if we were to draw a curve of a data set and consider the gradient of those
>> values, those values may have come from the same source. Otherwise the
>> gradient value we are calculating might not be correct.
>>
>> Thanks
>>
>> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Imagine we put the values in a graph (value against time). The gradient
>>> is the slope of the line. When we draw the line via two very near dots, we
>>> get a high slope/gradient even for a low increase of the values. This
>>> happens when we have only events which are occurred in very near time
>>> intervals.
>>>
>>> On the other hands, when events are distributed among long time period
>>> (are more counts in sample) we don't get this high gradient since the line
>>> goes via higher dots count where we get a reasonable value for gradient.
>>>
>>> As I understood the scenario, it is reasonable to make time gap equals
>>> to millisInSeconds in case time gap is very small.
>>>
>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>> I'd rather keep it simple Akila (this would complex the scenario of
>>> second derivative too) :-) Isn't it?
>>>
>>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>>> ravihansa@wso2.com> wrote:
>>>
>>>> Until t2 - t1 > 1
>>>>
>>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <
>>>> nirmal070125@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>>> ravihansa@wso2.com> wrote:
>>>>>
>>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>>
>>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>>
>>>>>
>>>>> What would be the sufficient value? Theoretically in gradient case
>>>>> it's 2, in second derivative case it's 3. And we're doing that already.
>>>>>
>>>>>>
>>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>>> samples.
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>>> taking the difference*.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Raj: I agree, in this scenario it makes no difference, however
>>>>>>>>> when we are applying a fomular the unit conversion should happen prior to
>>>>>>>>> applying values.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>>>>> division operation to make each operands to seconds, then since both are
>>>>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Still I see a problem with this correction, take the same example:
>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>
>>>>>>>>> *Manual calulation:*
>>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>>>>> 0.005 sec
>>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>>
>>>>>>>>> *According to the current fix:*
>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>> millisecondsForASecond;
>>>>>>>>> double gradient = 0.0;
>>>>>>>>> if (tGap > 0) {
>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>>
>>>>>>>>
>>>>>>>> IMO, this is a sufficient solution.
>>>>>>>>
>>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>>
>>>>>>>> We are taking only first and last values in a window for gradient
>>>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>>
>>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>>> deal with samples which are not good enough.
>>>>>>>>
>>>>>>>> By previous formula, we get -1000. This happened because the sample
>>>>>>>> is not good enough (may be there are few events). Fix makes it to -5.
>>>>>>>>
>>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <gayang@wso2.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>>>>
>>>>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>>
>>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You got it! thanks.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Gayan
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>>> correct here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As I see we are taking the difference of two timestamp values
>>>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>>
>>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>>
>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>>
>>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but
>>>>>>>>>>>> the result's unit will be different.
>>>>>>>>>>>>
>>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>>
>>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>>
>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2
>>>>>>>>>>>>>> - t1)*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug
>>>>>>>>>>>>>> logs;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157 hash:
>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, as you can see the reason behind a large value is when
>>>>>>>>>>>>>> the time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>>> *____
>>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>>> *|
>>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>>> *|____
>>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge
>>>>>>>>>>>>>>> agent also right?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1]
>>>>>>>>>>>>>>>>>> INFO EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too.
>>>>>>>>>>>>>>>>> This is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current
>>>>>>>>>>>>>>>>>> execution plans it took networkPartitionId for the calculation. IMO we need
>>>>>>>>>>>>>>>>>> to update the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in
>>>>>>>>>>>>>>>>>>>> the previous mail.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There
>>>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and
>>>>>>>>>>>>>>>>>>>>>> see what are the load average values sent from the agent. If those values
>>>>>>>>>>>>>>>>>>>>>> are normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar
>>>>>>>>>>>>>>>>>>>>>>> Rajaratnam <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent.
>>>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>
>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Raj
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>> Technical Lead
>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>> Nirmal
>>>>>>>>>>
>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>
>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Imesh Gunaratne
>>>>>>>>>
>>>>>>>>> Technical Lead, WSO2
>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Raj
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Raj
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Akila Ravihansa Perera
>>>>>> Software Engineer, WSO2
>>>>>>
>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nirmal
>>>>>
>>>>> Nirmal Fernando.
>>>>> PPMC Member & Committer of Apache Stratos,
>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Akila Ravihansa Perera
>>>> Software Engineer, WSO2
>>>>
>>>> Blog: http://ravihansa3000.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Lahiru Sandaruwan <la...@wso2.com>.
On Thu, Nov 6, 2014 at 7:15 PM, Lahiru Sandaruwan <la...@wso2.com> wrote:
> Hi all,
>
> Very interesting discussion and thanks Nirmal for initiating this!
>
> I also did some research on this.
> I think best approach to find gradient and second derivative is to "curve
> fitting". Refer "Locally weighted linear regression" section at [1]. We
> should be calculating gradient and second derivative by differentiating the
> equation of fitted line.
>
> We would need a second degree polynomial fitter for this, where we can use
> Apache commons math library for this. Refer the sample at [2], we need to
> run this with degree 2 instead of 3.
>
> E.g.
> So if get degree 2 polynomial fitter, we will have an equation like below
> where value(v) is our statistic value, time(t) is the time of event,
> gradient(g), and second derivative(sd).
>
> Equation we get from received events,
> v = a*t*t + b*t + c
>
> g = 2*a*t
> sd = 2*a
>
> We can get *g* by applying the median of the time stamp sample.
>
> I have run a sample successfully. I can complete the work on CEP extension
> if you guys agree.
>
> In addition to this, we should also restrict the sample size as Akila
> suggested. Ideally we should receive 6 events per minute for memberwise
> stats and "6 * no. of instances" for clusterwise stats. We should put a
> constraint with this much of percentage(say 80%) should be there to
> calculate *g *and* sd.*
>
> Also we should not calculated gradient of stats sent from different
> members in clusterwise scenario as Imesh mentioned. That causes this type
> of small time gap issues.
>
I have thought of a solution for this. Also most importantly the solution i
propose suits very well for the prediction requirement at Autoscaler, than
what we have now. Please go through and give feedback.
*CEP's task*
- Find memberwise curves that fits events received in 1 minute(or
required) window at CEP
- We send the parameters of fitted line(a, b, and c in above equation)
with the timestamp of last event(T) in the window, to Autoscaler
*Autoscaler's task*
- Autoscaler use *v = a*t*t + b*t + c *function to predict the value in
any timestamp from the last timestamp
E.g. Say we need to find the value(v) after 1 minute(assuming we carried
all the calculations in milliseconds),
v = a * (T+60000) * (T+60000) + b * (T+60000) + c
- So we have memberwise predictions and we can find clusterwise
prediction by averaging all the memberwise values.
Please send your thoughts.
Thanks.
> Thanks.
>
> [1] http://cs229.stanford.edu/notes/cs229-notes1.pdf
> [2] http://commons.apache.org/proper/commons-math/userguide/fitting.html
>
> On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Raj: Thanks for the clarification, it was my misunderstanding.
>>
>> I did further analysis on this issue, seems like the root cause of this
>> problem resides somewhere else.
>>
>> Currently we have set the cartridge agent's statistics publishing
>> interval to 15 seconds. If so how do we receive series of events within a
>> second? Seems like we are considering statistics events coming from
>> different members for calculating the gradient. This might not be correct,
>> if we were to draw a curve of a data set and consider the gradient of those
>> values, those values may have come from the same source. Otherwise the
>> gradient value we are calculating might not be correct.
>>
>> Thanks
>>
>> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Imagine we put the values in a graph (value against time). The gradient
>>> is the slope of the line. When we draw the line via two very near dots, we
>>> get a high slope/gradient even for a low increase of the values. This
>>> happens when we have only events which are occurred in very near time
>>> intervals.
>>>
>>> On the other hands, when events are distributed among long time period
>>> (are more counts in sample) we don't get this high gradient since the line
>>> goes via higher dots count where we get a reasonable value for gradient.
>>>
>>> As I understood the scenario, it is reasonable to make time gap equals
>>> to millisInSeconds in case time gap is very small.
>>>
>>> Touched, not typed. Erroneous words are a feature, not a typo.
>>> I'd rather keep it simple Akila (this would complex the scenario of
>>> second derivative too) :-) Isn't it?
>>>
>>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>>> ravihansa@wso2.com> wrote:
>>>
>>>> Until t2 - t1 > 1
>>>>
>>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <
>>>> nirmal070125@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>>> ravihansa@wso2.com> wrote:
>>>>>
>>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>>
>>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>>
>>>>>
>>>>> What would be the sufficient value? Theoretically in gradient case
>>>>> it's 2, in second derivative case it's 3. And we're doing that already.
>>>>>
>>>>>>
>>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>>> samples.
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>>> taking the difference*.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Raj: I agree, in this scenario it makes no difference, however
>>>>>>>>> when we are applying a fomular the unit conversion should happen prior to
>>>>>>>>> applying values.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>>>>> division operation to make each operands to seconds, then since both are
>>>>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Still I see a problem with this correction, take the same example:
>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>
>>>>>>>>> *Manual calulation:*
>>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>>>>> 0.005 sec
>>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>>
>>>>>>>>> *According to the current fix:*
>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>> millisecondsForASecond;
>>>>>>>>> double gradient = 0.0;
>>>>>>>>> if (tGap > 0) {
>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>>
>>>>>>>>
>>>>>>>> IMO, this is a sufficient solution.
>>>>>>>>
>>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>>
>>>>>>>> We are taking only first and last values in a window for gradient
>>>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>>
>>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>>> deal with samples which are not good enough.
>>>>>>>>
>>>>>>>> By previous formula, we get -1000. This happened because the sample
>>>>>>>> is not good enough (may be there are few events). Fix makes it to -5.
>>>>>>>>
>>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <gayang@wso2.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>>>>
>>>>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>>
>>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You got it! thanks.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Gayan
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>>> correct here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As I see we are taking the difference of two timestamp values
>>>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>>
>>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>>
>>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>>
>>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but
>>>>>>>>>>>> the result's unit will be different.
>>>>>>>>>>>>
>>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>>
>>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>>
>>>>>>>>>>>> wdyt?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2
>>>>>>>>>>>>>> - t1)*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug
>>>>>>>>>>>>>> logs;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157 hash:
>>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, as you can see the reason behind a large value is when
>>>>>>>>>>>>>> the time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>>> *____
>>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>>> *|
>>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>>> *|____
>>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge
>>>>>>>>>>>>>>> agent also right?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1]
>>>>>>>>>>>>>>>>>> INFO EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too.
>>>>>>>>>>>>>>>>> This is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current
>>>>>>>>>>>>>>>>>> execution plans it took networkPartitionId for the calculation. IMO we need
>>>>>>>>>>>>>>>>>> to update the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in
>>>>>>>>>>>>>>>>>>>> the previous mail.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There
>>>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and
>>>>>>>>>>>>>>>>>>>>>> see what are the load average values sent from the agent. If those values
>>>>>>>>>>>>>>>>>>>>>> are normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar
>>>>>>>>>>>>>>>>>>>>>>> Rajaratnam <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent.
>>>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>
>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Raj
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>> Technical Lead
>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>> Nirmal
>>>>>>>>>>
>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>
>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Imesh Gunaratne
>>>>>>>>>
>>>>>>>>> Technical Lead, WSO2
>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Raj
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Raj
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Akila Ravihansa Perera
>>>>>> Software Engineer, WSO2
>>>>>>
>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nirmal
>>>>>
>>>>> Nirmal Fernando.
>>>>> PPMC Member & Committer of Apache Stratos,
>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Akila Ravihansa Perera
>>>> Software Engineer, WSO2
>>>>
>>>> Blog: http://ravihansa3000.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Lahiru Sandaruwan <la...@wso2.com>.
Hi all,
Very interesting discussion and thanks Nirmal for initiating this!
I also did some research on this.
I think best approach to find gradient and second derivative is to "curve
fitting". Refer "Locally weighted linear regression" section at [1]. We
should be calculating gradient and second derivative by differentiating the
equation of fitted line.
We would need a second degree polynomial fitter for this, where we can use
Apache commons math library for this. Refer the sample at [2], we need to
run this with degree 2 instead of 3.
E.g.
So if get degree 2 polynomial fitter, we will have an equation like below
where value(v) is our statistic value, time(t) is the time of event,
gradient(g), and second derivative(sd).
Equation we get from received events,
v = a*t*t + b*t + c
g = 2*a*t
sd = 2*a
We can get *g* by applying the median of the time stamp sample.
I have run a sample successfully. I can complete the work on CEP extension
if you guys agree.
In addition to this, we should also restrict the sample size as Akila
suggested. Ideally we should receive 6 events per minute for memberwise
stats and "6 * no. of instances" for clusterwise stats. We should put a
constraint with this much of percentage(say 80%) should be there to
calculate *g *and* sd.*
Also we should not calculated gradient of stats sent from different members
in clusterwise scenario as Imesh mentioned. That causes this type of small
time gap issues.
Thanks.
[1] http://cs229.stanford.edu/notes/cs229-notes1.pdf
[2] http://commons.apache.org/proper/commons-math/userguide/fitting.html
On Thu, Nov 6, 2014 at 6:41 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Raj: Thanks for the clarification, it was my misunderstanding.
>
> I did further analysis on this issue, seems like the root cause of this
> problem resides somewhere else.
>
> Currently we have set the cartridge agent's statistics publishing interval
> to 15 seconds. If so how do we receive series of events within a second?
> Seems like we are considering statistics events coming from different
> members for calculating the gradient. This might not be correct, if we were
> to draw a curve of a data set and consider the gradient of those values,
> those values may have come from the same source. Otherwise the gradient
> value we are calculating might not be correct.
>
> Thanks
>
> On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi,
>>
>> Imagine we put the values in a graph (value against time). The gradient
>> is the slope of the line. When we draw the line via two very near dots, we
>> get a high slope/gradient even for a low increase of the values. This
>> happens when we have only events which are occurred in very near time
>> intervals.
>>
>> On the other hands, when events are distributed among long time period
>> (are more counts in sample) we don't get this high gradient since the line
>> goes via higher dots count where we get a reasonable value for gradient.
>>
>> As I understood the scenario, it is reasonable to make time gap equals to
>> millisInSeconds in case time gap is very small.
>>
>> Touched, not typed. Erroneous words are a feature, not a typo.
>> I'd rather keep it simple Akila (this would complex the scenario of
>> second derivative too) :-) Isn't it?
>>
>> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <
>> ravihansa@wso2.com> wrote:
>>
>>> Until t2 - t1 > 1
>>>
>>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <nirmal070125@gmail.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>>> ravihansa@wso2.com> wrote:
>>>>
>>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>>
>>>>> Perhaps as an alternative solution, window processor can be set to
>>>>> wait until sufficient events have arrived to calculate the gradient?
>>>>>
>>>>
>>>> What would be the sufficient value? Theoretically in gradient case it's
>>>> 2, in second derivative case it's 3. And we're doing that already.
>>>>
>>>>>
>>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>> In other words, fix reduces the risks which are coming from bad
>>>>>> samples.
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>> taking the difference*.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Raj: I agree, in this scenario it makes no difference, however when
>>>>>>>> we are applying a fomular the unit conversion should happen prior to
>>>>>>>> applying values.
>>>>>>>>
>>>>>>>
>>>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>>>> division operation to make each operands to seconds, then since both are
>>>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>>
>>>>>>>
>>>>>>>> Still I see a problem with this correction, take the same example:
>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>
>>>>>>>> *Manual calulation:*
>>>>>>>> The value difference: 7 - 12 = -5
>>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>>>> 0.005 sec
>>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>>
>>>>>>>> *According to the current fix:*
>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>> millisecondsForASecond;
>>>>>>>> double gradient = 0.0;
>>>>>>>> if (tGap > 0) {
>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>> }
>>>>>>>>
>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>>
>>>>>>>
>>>>>>> IMO, this is a sufficient solution.
>>>>>>>
>>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>>
>>>>>>> We are taking only first and last values in a window for gradient
>>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>>
>>>>>>> If there are few events in that window(say only 2), they can be
>>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>>> in the time interval between two events. This fix is really important to
>>>>>>> deal with samples which are not good enough.
>>>>>>>
>>>>>>> By previous formula, we get -1000. This happened because the sample
>>>>>>> is not good enough (may be there are few events). Fix makes it to -5.
>>>>>>>
>>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>>
>>>>>>>> Nirmal: Can you please justify this?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>>>
>>>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>>
>>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You got it! thanks.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Gayan
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <
>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>>
>>>>>>>>>>>> I do not think the milliseconds to seconds convertion is
>>>>>>>>>>>> correct here.
>>>>>>>>>>>>
>>>>>>>>>>>> As I see we are taking the difference of two timestamp values
>>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000
>>>>>>>>>>> and taking the difference*.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>>
>>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>>
>>>>>>>>>>>> *According previous code:*
>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>>
>>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1
>>>>>>>>>>>> : millisecondsForASecond;
>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>>
>>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>>> gradient: -1
>>>>>>>>>>>>
>>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>>
>>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>>
>>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>>
>>>>>>>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>>>>>>>> result's unit will be different.
>>>>>>>>>>>
>>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your
>>>>>>>>>>> case, you will get *stats/milliseconds*
>>>>>>>>>>>
>>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use
>>>>>>>>>>> *seconds* for the calculation.
>>>>>>>>>>>
>>>>>>>>>>> wdyt?
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>>
>>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>>
>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2
>>>>>>>>>>>>> - t1)*
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>>
>>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash:
>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>>>>>>> 12.0 *Time Gap: 5* t1: 1415213232152 t2: 1415213232157 hash:
>>>>>>>>>>>>> 155426542
>>>>>>>>>>>>>
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>>>
>>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, as you can see the reason behind a large value is when the
>>>>>>>>>>>>> time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>>> *____
>>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>>> *|
>>>>>>>>>>>>> yes?*
>>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>>> *|____
>>>>>>>>>>>>> 1000*
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent
>>>>>>>>>>>>>> also right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too. This
>>>>>>>>>>>>>>>> is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current execution
>>>>>>>>>>>>>>>>> plans it took networkPartitionId for the calculation. IMO we need to update
>>>>>>>>>>>>>>>>> the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in
>>>>>>>>>>>>>>>>>>> the previous mail.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There
>>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see
>>>>>>>>>>>>>>>>>>>>> what are the load average values sent from the agent. If those values are
>>>>>>>>>>>>>>>>>>>>> normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>> <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent.
>>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>
>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Raj
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>> Technical Lead
>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Nirmal
>>>>>>>>>
>>>>>>>>> Nirmal Fernando.
>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>
>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Imesh Gunaratne
>>>>>>>>
>>>>>>>> Technical Lead, WSO2
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Raj
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Raj
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Akila Ravihansa Perera
>>>>> Software Engineer, WSO2
>>>>>
>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Nirmal
>>>>
>>>> Nirmal Fernando.
>>>> PPMC Member & Committer of Apache Stratos,
>>>> Senior Software Engineer, WSO2 Inc.
>>>>
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Akila Ravihansa Perera
>>> Software Engineer, WSO2
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahirus@wso2.com blog: http://lahiruwrites.blogspot.com/
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Imesh Gunaratne <im...@apache.org>.
Raj: Thanks for the clarification, it was my misunderstanding.
I did further analysis on this issue, seems like the root cause of this
problem resides somewhere else.
Currently we have set the cartridge agent's statistics publishing interval
to 15 seconds. If so how do we receive series of events within a second?
Seems like we are considering statistics events coming from different
members for calculating the gradient. This might not be correct, if we were
to draw a curve of a data set and consider the gradient of those values,
those values may have come from the same source. Otherwise the gradient
value we are calculating might not be correct.
Thanks
On Thu, Nov 6, 2014 at 2:05 PM, Udara Liyanage <ud...@wso2.com> wrote:
> Hi,
>
> Imagine we put the values in a graph (value against time). The gradient is
> the slope of the line. When we draw the line via two very near dots, we get
> a high slope/gradient even for a low increase of the values. This happens
> when we have only events which are occurred in very near time intervals.
>
> On the other hands, when events are distributed among long time period
> (are more counts in sample) we don't get this high gradient since the line
> goes via higher dots count where we get a reasonable value for gradient.
>
> As I understood the scenario, it is reasonable to make time gap equals to
> millisInSeconds in case time gap is very small.
>
> Touched, not typed. Erroneous words are a feature, not a typo.
> I'd rather keep it simple Akila (this would complex the scenario of second
> derivative too) :-) Isn't it?
>
> On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <ravihansa@wso2.com
> > wrote:
>
>> Until t2 - t1 > 1
>>
>> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <ni...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>>> ravihansa@wso2.com> wrote:
>>>
>>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>>
>>>> Perhaps as an alternative solution, window processor can be set to wait
>>>> until sufficient events have arrived to calculate the gradient?
>>>>
>>>
>>> What would be the sufficient value? Theoretically in gradient case it's
>>> 2, in second derivative case it's 3. And we're doing that already.
>>>
>>>>
>>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>> In other words, fix reduces the risks which are coming from bad
>>>>> samples.
>>>>>
>>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>> taking the difference*.
>>>>>>>>
>>>>>>>
>>>>>>> Raj: I agree, in this scenario it makes no difference, however when
>>>>>>> we are applying a fomular the unit conversion should happen prior to
>>>>>>> applying values.
>>>>>>>
>>>>>>
>>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>>> division operation to make each operands to seconds, then since both are
>>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>>
>>>>>>
>>>>>>> Still I see a problem with this correction, take the same example:
>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>
>>>>>>> *Manual calulation:*
>>>>>>> The value difference: 7 - 12 = -5
>>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>>> 0.005 sec
>>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>>
>>>>>>> *According to the current fix:*
>>>>>>> long millisecondsForASecond = 1000;
>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>> millisecondsForASecond;
>>>>>>> double gradient = 0.0;
>>>>>>> if (tGap > 0) {
>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>> millisecondsForASecond) / tGap;
>>>>>>> }
>>>>>>>
>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>>
>>>>>>
>>>>>> IMO, this is a sufficient solution.
>>>>>>
>>>>>> Let me explain how I see Nirmal's solution.
>>>>>>
>>>>>> We are taking only first and last values in a window for gradient
>>>>>> calculation. If there are more events in that window, and if that is a good
>>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>>
>>>>>> If there are few events in that window(say only 2), they can be
>>>>>> really nearer. So there is a possibility that tGap can take a value near to
>>>>>> 0. In that case gradient will be a very big value. Hence, if the tGap is
>>>>>> very low (less than 1 second), then we can assume that the first and last
>>>>>> events actually occurred in 1 second gap. The reason why tGap is less than
>>>>>> 0 is, there are only few events in that window, and that sample is not good
>>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>>> in the time interval between two events. This fix is really important to
>>>>>> deal with samples which are not good enough.
>>>>>>
>>>>>> By previous formula, we get -1000. This happened because the sample
>>>>>> is not good enough (may be there are few events). Fix makes it to -5.
>>>>>>
>>>>>> So my understanding is that the solution is good enough for now.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>>
>>>>>>> Nirmal: Can you please justify this?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>>
>>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>>
>>>>>>>>> I think that's what Nirmal doing here by making the lower
>>>>>>>>> threshold as "millisecondsForASecond"
>>>>>>>>>
>>>>>>>>
>>>>>>>> You got it! thanks.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>> millisecondsForASecond;
>>>>>>>>> double gradient = 0.0;
>>>>>>>>> if (tGap > 0) {
>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Gayan
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <imesh@apache.org
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>>
>>>>>>>>>>> I do not think the milliseconds to seconds convertion is correct
>>>>>>>>>>> here.
>>>>>>>>>>>
>>>>>>>>>>> As I see we are taking the difference of two timestamp values
>>>>>>>>>>> and then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>>> taking the difference*.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>>
>>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>>
>>>>>>>>>>> *According previous code:*
>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>>
>>>>>>>>>>> *According to your fix:*
>>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>> gradient: -5.0
>>>>>>>>>>>
>>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>>> gradient: -1
>>>>>>>>>>>
>>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>>
>>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>>
>>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>>> gradient: -1.0
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>>
>>>>>>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>>>>>>> result's unit will be different.
>>>>>>>>>>
>>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your case,
>>>>>>>>>> you will get *stats/milliseconds*
>>>>>>>>>>
>>>>>>>>>> And AS prediction is based on *seconds*, CEP should use *seconds*
>>>>>>>>>> for the calculation.
>>>>>>>>>>
>>>>>>>>>> wdyt?
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>
>>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>>
>>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>>
>>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>>
>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>>
>>>>>>>>>>>> Hence,
>>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 -
>>>>>>>>>>>> t1)*
>>>>>>>>>>>>
>>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>>
>>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>>
>>>>>>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>>>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>>
>>>>>>>>>>>> ===================================================================
>>>>>>>>>>>>
>>>>>>>>>>>> So, as you can see the reason behind a large value is when the
>>>>>>>>>>>> time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>>
>>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>>
>>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>>> *____
>>>>>>>>>>>> t2 - t1*
>>>>>>>>>>>> *|
>>>>>>>>>>>> yes?*
>>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>>> *|____
>>>>>>>>>>>> 1000*
>>>>>>>>>>>>
>>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and
>>>>>>>>>>>> second derivative of load average
>>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes it has. But it's very easy to find the values from the
>>>>>>>>>>>> agent with the time stamp. Then we can search that time stamp in the CEP
>>>>>>>>>>>> trace logs and compare the summarized values.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika
>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent
>>>>>>>>>>>>> also right?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is happening intermittently. Experienced now too. This
>>>>>>>>>>>>>>> is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current execution
>>>>>>>>>>>>>>>> plans it took networkPartitionId for the calculation. IMO we need to update
>>>>>>>>>>>>>>>> the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in
>>>>>>>>>>>>>>>>>> the previous mail.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There
>>>>>>>>>>>>>>>>>>> are traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see
>>>>>>>>>>>>>>>>>>>> what are the load average values sent from the agent. If those values are
>>>>>>>>>>>>>>>>>>>> normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>> <ra...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent.
>>>>>>>>>>>>>>>>>>>>>> And faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>
>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Nirmal
>>>>>>>>>>>>
>>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>>
>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>
>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Raj
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Gayan Gunarathne
>>>>>>>>> Technical Lead
>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Nirmal
>>>>>>>>
>>>>>>>> Nirmal Fernando.
>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>
>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Imesh Gunaratne
>>>>>>>
>>>>>>> Technical Lead, WSO2
>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Raj
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Raj
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Akila Ravihansa Perera
>>>> Software Engineer, WSO2
>>>>
>>>> Blog: http://ravihansa3000.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> Akila Ravihansa Perera
>> Software Engineer, WSO2
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Udara Liyanage <ud...@wso2.com>.
Hi,
Imagine we put the values in a graph (value against time). The gradient is
the slope of the line. When we draw the line via two very near dots, we get
a high slope/gradient even for a low increase of the values. This happens
when we have only events which are occurred in very near time intervals.
On the other hands, when events are distributed among long time period (are
more counts in sample) we don't get this high gradient since the line goes
via higher dots count where we get a reasonable value for gradient.
As I understood the scenario, it is reasonable to make time gap equals to
millisInSeconds in case time gap is very small.
Touched, not typed. Erroneous words are a feature, not a typo.
I'd rather keep it simple Akila (this would complex the scenario of second
derivative too) :-) Isn't it?
On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <ra...@wso2.com>
wrote:
> Until t2 - t1 > 1
>
> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
>>
>>
>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>> ravihansa@wso2.com> wrote:
>>
>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>
>>> Perhaps as an alternative solution, window processor can be set to wait
>>> until sufficient events have arrived to calculate the gradient?
>>>
>>
>> What would be the sufficient value? Theoretically in gradient case it's
>> 2, in second derivative case it's 3. And we're doing that already.
>>
>>>
>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>> In other words, fix reduces the risks which are coming from bad samples.
>>>>
>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>> taking the difference*.
>>>>>>>
>>>>>>
>>>>>> Raj: I agree, in this scenario it makes no difference, however when
>>>>>> we are applying a fomular the unit conversion should happen prior to
>>>>>> applying values.
>>>>>>
>>>>>
>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>> division operation to make each operands to seconds, then since both are
>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>
>>>>>
>>>>>> Still I see a problem with this correction, take the same example:
>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>
>>>>>> *Manual calulation:*
>>>>>> The value difference: 7 - 12 = -5
>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>> 0.005 sec
>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>
>>>>>> *According to the current fix:*
>>>>>> long millisecondsForASecond = 1000;
>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>> millisecondsForASecond;
>>>>>> double gradient = 0.0;
>>>>>> if (tGap > 0) {
>>>>>> gradient = ((lastVal - firstVal) *
>>>>>> millisecondsForASecond) / tGap;
>>>>>> }
>>>>>>
>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>
>>>>>
>>>>> IMO, this is a sufficient solution.
>>>>>
>>>>> Let me explain how I see Nirmal's solution.
>>>>>
>>>>> We are taking only first and last values in a window for gradient
>>>>> calculation. If there are more events in that window, and if that is a good
>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>
>>>>> If there are few events in that window(say only 2), they can be really
>>>>> nearer. So there is a possibility that tGap can take a value near to 0. In
>>>>> that case gradient will be a very big value. Hence, if the tGap is very low
>>>>> (less than 1 second), then we can assume that the first and last events
>>>>> actually occurred in 1 second gap. The reason why tGap is less than 0 is,
>>>>> there are only few events in that window, and that sample is not good
>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>> in the time interval between two events. This fix is really important to
>>>>> deal with samples which are not good enough.
>>>>>
>>>>> By previous formula, we get -1000. This happened because the sample is
>>>>> not good enough (may be there are few events). Fix makes it to -5.
>>>>>
>>>>> So my understanding is that the solution is good enough for now.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>>
>>>>>> Nirmal: Can you please justify this?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>
>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>
>>>>>>>> I think that's what Nirmal doing here by making the lower threshold
>>>>>>>> as "millisecondsForASecond"
>>>>>>>>
>>>>>>>
>>>>>>> You got it! thanks.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>> millisecondsForASecond;
>>>>>>>> double gradient = 0.0;
>>>>>>>> if (tGap > 0) {
>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Gayan
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>
>>>>>>>>>> I do not think the milliseconds to seconds convertion is correct
>>>>>>>>>> here.
>>>>>>>>>>
>>>>>>>>>> As I see we are taking the difference of two timestamp values and
>>>>>>>>>> then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>> taking the difference*.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>
>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>
>>>>>>>>>> *According previous code:*
>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>
>>>>>>>>>> *According to your fix:*
>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>> gradient: -5.0
>>>>>>>>>>
>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>> gradient: -1
>>>>>>>>>>
>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>
>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>
>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>> gradient: -1.0
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>
>>>>>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>>>>>> result's unit will be different.
>>>>>>>>>
>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your case,
>>>>>>>>> you will get *stats/milliseconds*
>>>>>>>>>
>>>>>>>>> And AS prediction is based on *seconds*, CEP should use *seconds*
>>>>>>>>> for the calculation.
>>>>>>>>>
>>>>>>>>> wdyt?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>
>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>
>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>
>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>
>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>
>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>
>>>>>>>>>>> Hence,
>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 -
>>>>>>>>>>> t1)*
>>>>>>>>>>>
>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>
>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>
>>>>>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ===================================================================
>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>>>>>>
>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>>>>>>
>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>
>>>>>>>>>>> ===================================================================
>>>>>>>>>>>
>>>>>>>>>>> So, as you can see the reason behind a large value is when the
>>>>>>>>>>> time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>
>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>
>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>> *____
>>>>>>>>>>> t2 - t1*
>>>>>>>>>>> *| yes?*
>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>> *|____
>>>>>>>>>>> 1000*
>>>>>>>>>>>
>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>>>>>>>> derivative of load average
>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>
>>>>>>>>>>> Yes it has. But it's very easy to find the values from the agent
>>>>>>>>>>> with the time stamp. Then we can search that time stamp in the CEP trace
>>>>>>>>>>> logs and compare the summarized values.
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte
>>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent
>>>>>>>>>>>> also right?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is happening intermittently. Experienced now too. This
>>>>>>>>>>>>>> is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current execution
>>>>>>>>>>>>>>> plans it took networkPartitionId for the calculation. IMO we need to update
>>>>>>>>>>>>>>> the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>>>>>>>> previous mail.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see
>>>>>>>>>>>>>>>>>>> what are the load average values sent from the agent. If those values are
>>>>>>>>>>>>>>>>>>> normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And
>>>>>>>>>>>>>>>>>>>>> faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Nirmal
>>>>>>>>>>>
>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>
>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>
>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Raj
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Gayan Gunarathne
>>>>>>>> Technical Lead
>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Nirmal
>>>>>>>
>>>>>>> Nirmal Fernando.
>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>
>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Raj
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Raj
>>>>
>>>
>>>
>>>
>>> --
>>> Akila Ravihansa Perera
>>> Software Engineer, WSO2
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Akila Ravihansa Perera
> Software Engineer, WSO2
>
> Blog: http://ravihansa3000.blogspot.com
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
I'd rather keep it simple Akila (this would complex the scenario of second
derivative too) :-) Isn't it?
On Thu, Nov 6, 2014 at 9:03 AM, Akila Ravihansa Perera <ra...@wso2.com>
wrote:
> Until t2 - t1 > 1
>
> On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
>>
>>
>> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <
>> ravihansa@wso2.com> wrote:
>>
>>> Thanks for the explanation, Nirmal. It makes sense now.
>>>
>>> Perhaps as an alternative solution, window processor can be set to wait
>>> until sufficient events have arrived to calculate the gradient?
>>>
>>
>> What would be the sufficient value? Theoretically in gradient case it's
>> 2, in second derivative case it's 3. And we're doing that already.
>>
>>>
>>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>> In other words, fix reduces the risks which are coming from bad samples.
>>>>
>>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>> taking the difference*.
>>>>>>>
>>>>>>
>>>>>> Raj: I agree, in this scenario it makes no difference, however when
>>>>>> we are applying a fomular the unit conversion should happen prior to
>>>>>> applying values.
>>>>>>
>>>>>
>>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>>> division operation to make it in seconds. On the other hand, we can do a
>>>>> division operation to make each operands to seconds, then since both are
>>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>>
>>>>>
>>>>>> Still I see a problem with this correction, take the same example:
>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>
>>>>>> *Manual calulation:*
>>>>>> The value difference: 7 - 12 = -5
>>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>>> 0.005 sec
>>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>>
>>>>>> *According to the current fix:*
>>>>>> long millisecondsForASecond = 1000;
>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>> millisecondsForASecond;
>>>>>> double gradient = 0.0;
>>>>>> if (tGap > 0) {
>>>>>> gradient = ((lastVal - firstVal) *
>>>>>> millisecondsForASecond) / tGap;
>>>>>> }
>>>>>>
>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>>
>>>>>
>>>>> IMO, this is a sufficient solution.
>>>>>
>>>>> Let me explain how I see Nirmal's solution.
>>>>>
>>>>> We are taking only first and last values in a window for gradient
>>>>> calculation. If there are more events in that window, and if that is a good
>>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>>
>>>>> If there are few events in that window(say only 2), they can be really
>>>>> nearer. So there is a possibility that tGap can take a value near to 0. In
>>>>> that case gradient will be a very big value. Hence, if the tGap is very low
>>>>> (less than 1 second), then we can assume that the first and last events
>>>>> actually occurred in 1 second gap. The reason why tGap is less than 0 is,
>>>>> there are only few events in that window, and that sample is not good
>>>>> enough. But this a rare situation, it can only happen once in a while.
>>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>>> in the time interval between two events. This fix is really important to
>>>>> deal with samples which are not good enough.
>>>>>
>>>>> By previous formula, we get -1000. This happened because the sample is
>>>>> not good enough (may be there are few events). Fix makes it to -5.
>>>>>
>>>>> So my understanding is that the solution is good enough for now.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>>
>>>>>> Nirmal: Can you please justify this?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> AFAIU this is not related with the gradient calculation formula
>>>>>>>> that we used. But It is related with the how we calculate gradient when the
>>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>>
>>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>>
>>>>>>>> I think that's what Nirmal doing here by making the lower threshold
>>>>>>>> as "millisecondsForASecond"
>>>>>>>>
>>>>>>>
>>>>>>> You got it! thanks.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>> millisecondsForASecond;
>>>>>>>> double gradient = 0.0;
>>>>>>>> if (tGap > 0) {
>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Gayan
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Nirmal,
>>>>>>>>>>
>>>>>>>>>> I do not think the milliseconds to seconds convertion is correct
>>>>>>>>>> here.
>>>>>>>>>>
>>>>>>>>>> As I see we are taking the difference of two timestamp values and
>>>>>>>>>> then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>>> taking the difference*.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>>
>>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>>
>>>>>>>>>> *According previous code:*
>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>> gradient: -1000.0
>>>>>>>>>>
>>>>>>>>>> *According to your fix:*
>>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>>> millisecondsForASecond;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>> gradient: -5.0
>>>>>>>>>>
>>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>>> gradient: -1
>>>>>>>>>>
>>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>>
>>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>>
>>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>>> double gradient = 0.0;
>>>>>>>>>> if (tGap > 0) {
>>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>>> gradient: -1.0
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>>
>>>>>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>>>>>> result's unit will be different.
>>>>>>>>>
>>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your case,
>>>>>>>>> you will get *stats/milliseconds*
>>>>>>>>>
>>>>>>>>> And AS prediction is based on *seconds*, CEP should use *seconds*
>>>>>>>>> for the calculation.
>>>>>>>>>
>>>>>>>>> wdyt?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>
>>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>>
>>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>>
>>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>>
>>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>>
>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>>
>>>>>>>>>>> Hence,
>>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 -
>>>>>>>>>>> t1)*
>>>>>>>>>>>
>>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>>
>>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>>
>>>>>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ===================================================================
>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>>>>>>
>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>>>>>>
>>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>>
>>>>>>>>>>> ===================================================================
>>>>>>>>>>>
>>>>>>>>>>> So, as you can see the reason behind a large value is when the
>>>>>>>>>>> time gap between two subjected events is less than 1 second. This could
>>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>>> when there are less number of events.
>>>>>>>>>>>
>>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>>
>>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>>> *____
>>>>>>>>>>> t2 - t1*
>>>>>>>>>>> *| yes?*
>>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>>> *|____
>>>>>>>>>>> 1000*
>>>>>>>>>>>
>>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>>>>>>>> derivative of load average
>>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>
>>>>>>>>>>> Yes it has. But it's very easy to find the values from the agent
>>>>>>>>>>> with the time stamp. Then we can search that time stamp in the CEP trace
>>>>>>>>>>> logs and compare the summarized values.
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte
>>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent
>>>>>>>>>>>> also right?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is happening intermittently. Experienced now too. This
>>>>>>>>>>>>>> is a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current execution
>>>>>>>>>>>>>>> plans it took networkPartitionId for the calculation. IMO we need to update
>>>>>>>>>>>>>>> the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>>>>>>>> previous mail.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see
>>>>>>>>>>>>>>>>>>> what are the load average values sent from the agent. If those values are
>>>>>>>>>>>>>>>>>>> normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And
>>>>>>>>>>>>>>>>>>>>> faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since
>>>>>>>>>>>>>>>>>>>>> containers are destroyed.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Nirmal
>>>>>>>>>>>
>>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>>
>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>
>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Raj
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Gayan Gunarathne
>>>>>>>> Technical Lead
>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Nirmal
>>>>>>>
>>>>>>> Nirmal Fernando.
>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>
>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Raj
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Raj
>>>>
>>>
>>>
>>>
>>> --
>>> Akila Ravihansa Perera
>>> Software Engineer, WSO2
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Akila Ravihansa Perera
> Software Engineer, WSO2
>
> Blog: http://ravihansa3000.blogspot.com
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Until t2 - t1 > 1
On Thu, Nov 6, 2014 at 12:50 AM, Nirmal Fernando <ni...@gmail.com>
wrote:
>
>
> On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <ravihansa@wso2.com
> > wrote:
>
>> Thanks for the explanation, Nirmal. It makes sense now.
>>
>> Perhaps as an alternative solution, window processor can be set to wait
>> until sufficient events have arrived to calculate the gradient?
>>
>
> What would be the sufficient value? Theoretically in gradient case it's 2,
> in second derivative case it's 3. And we're doing that already.
>
>>
>> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>> In other words, fix reduces the risks which are coming from bad samples.
>>>
>>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> No Imesh. There are no difference between *taking the the difference
>>>>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>>>>> difference*.
>>>>>>
>>>>>
>>>>> Raj: I agree, in this scenario it makes no difference, however when we
>>>>> are applying a fomular the unit conversion should happen prior to applying
>>>>> values.
>>>>>
>>>>
>>>> Not exactly Imesh. We can do operations if operands are in the same
>>>> units. So we can do subtract operation in milliseconds. And then we can do
>>>> division operation to make it in seconds. On the other hand, we can do a
>>>> division operation to make each operands to seconds, then since both are
>>>> seconds, we can do a subtract operation. So we can apply the formula if
>>>> operands are in same unit. I guess this is same for all scenarios.
>>>>
>>>>
>>>>> Still I see a problem with this correction, take the same example:
>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>
>>>>> *Manual calulation:*
>>>>> The value difference: 7 - 12 = -5
>>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 =
>>>>> 0.005 sec
>>>>> *Gradient = - 5/0.005 = -1000*
>>>>>
>>>>> *According to the current fix:*
>>>>> long millisecondsForASecond = 1000;
>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>> millisecondsForASecond;
>>>>> double gradient = 0.0;
>>>>> if (tGap > 0) {
>>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond)
>>>>> / tGap;
>>>>> }
>>>>>
>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>>
>>>>
>>>> IMO, this is a sufficient solution.
>>>>
>>>> Let me explain how I see Nirmal's solution.
>>>>
>>>> We are taking only first and last values in a window for gradient
>>>> calculation. If there are more events in that window, and if that is a good
>>>> sample, tGap will be taking some higher values than 1s. This is usual
>>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>>
>>>> If there are few events in that window(say only 2), they can be really
>>>> nearer. So there is a possibility that tGap can take a value near to 0. In
>>>> that case gradient will be a very big value. Hence, if the tGap is very low
>>>> (less than 1 second), then we can assume that the first and last events
>>>> actually occurred in 1 second gap. The reason why tGap is less than 0 is,
>>>> there are only few events in that window, and that sample is not good
>>>> enough. But this a rare situation, it can only happen once in a while.
>>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>>> in the time interval between two events. This fix is really important to
>>>> deal with samples which are not good enough.
>>>>
>>>> By previous formula, we get -1000. This happened because the sample is
>>>> not good enough (may be there are few events). Fix makes it to -5.
>>>>
>>>> So my understanding is that the solution is good enough for now.
>>>>
>>>> Thanks.
>>>>
>>>>>
>>>>> Nirmal: Can you please justify this?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>>> nirmal070125@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> AFAIU this is not related with the gradient calculation formula that
>>>>>>> we used. But It is related with the how we calculate gradient when the
>>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>>
>>>>>>> IMO we do have max and min scale up and down values. So based on
>>>>>>> that we can calculate the gradient on that value range.Simply if the
>>>>>>> denominator approaches zero result approaches to infinity.
>>>>>>>
>>>>>>> I think that's what Nirmal doing here by making the lower threshold
>>>>>>> as "millisecondsForASecond"
>>>>>>>
>>>>>>
>>>>>> You got it! thanks.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>> millisecondsForASecond;
>>>>>>> double gradient = 0.0;
>>>>>>> if (tGap > 0) {
>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>> millisecondsForASecond) / tGap;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Gayan
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Nirmal,
>>>>>>>>>
>>>>>>>>> I do not think the milliseconds to seconds convertion is correct
>>>>>>>>> here.
>>>>>>>>>
>>>>>>>>> As I see we are taking the difference of two timestamp values and
>>>>>>>>> then dividing it by 1000. The corect way might be to first divide each
>>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>>
>>>>>>>>
>>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>>> taking the difference*.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>>
>>>>>>>>> I did a quick test with the following sample:
>>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First
>>>>>>>>> val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>>
>>>>>>>>> *According previous code:*
>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>> double gradient = 0.0;
>>>>>>>>> if (tGap > 0) {
>>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>> gradient: -1000.0
>>>>>>>>>
>>>>>>>>> *According to your fix:*
>>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>>> millisecondsForASecond;
>>>>>>>>> double gradient = 0.0;
>>>>>>>>> if (tGap > 0) {
>>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>> gradient: -5.0
>>>>>>>>>
>>>>>>>>> *According to an online gradient calculator:*
>>>>>>>>> gradient: -1
>>>>>>>>>
>>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>>
>>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>>
>>>>>>>>> long tGap = t2 - t1;
>>>>>>>>> double gradient = 0.0;
>>>>>>>>> if (tGap > 0) {
>>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>>> gradient: -1.0
>>>>>>>>>
>>>>>>>>
>>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>>
>>>>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>>>>> result's unit will be different.
>>>>>>>>
>>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your case,
>>>>>>>> you will get *stats/milliseconds*
>>>>>>>>
>>>>>>>> And AS prediction is based on *seconds*, CEP should use *seconds*
>>>>>>>> for the calculation.
>>>>>>>>
>>>>>>>> wdyt?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Guys,
>>>>>>>>>>
>>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>>
>>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>>
>>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>>
>>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>>
>>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>>
>>>>>>>>>> Hence,
>>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 -
>>>>>>>>>> t1)*
>>>>>>>>>>
>>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>>
>>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>>
>>>>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ===================================================================
>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>>>>>
>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>>>>>
>>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>>>
>>>>>>>>>> ===================================================================
>>>>>>>>>>
>>>>>>>>>> So, as you can see the reason behind a large value is when the
>>>>>>>>>> time gap between two subjected events is less than 1 second. This could
>>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>>> when there are less number of events.
>>>>>>>>>>
>>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>>
>>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>>> *____ t2
>>>>>>>>>> - t1*
>>>>>>>>>> *| yes?*
>>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>>> *|____
>>>>>>>>>> 1000*
>>>>>>>>>>
>>>>>>>>>> I have tested this and works fine.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>>>>>>> derivative of load average
>>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Raj,
>>>>>>>>>>
>>>>>>>>>> Yes it has. But it's very easy to find the values from the agent
>>>>>>>>>> with the time stamp. Then we can search that time stamp in the CEP trace
>>>>>>>>>> logs and compare the summarized values.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte
>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>
>>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent
>>>>>>>>>>> also right?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go
>>>>>>>>>>>>>> though the logs I'm unable to find the high input values for load average.
>>>>>>>>>>>>>> But output have high values.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is happening intermittently. Experienced now too. This is
>>>>>>>>>>>>> a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also networkPartitionId is empty. In the current execution
>>>>>>>>>>>>>> plans it took networkPartitionId for the calculation. IMO we need to update
>>>>>>>>>>>>>> the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>>>>>>> previous mail.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see
>>>>>>>>>>>>>>>>>> what are the load average values sent from the agent. If those values are
>>>>>>>>>>>>>>>>>> normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And
>>>>>>>>>>>>>>>>>>>> faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers
>>>>>>>>>>>>>>>>>>>> are destroyed.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>
>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>> Nirmal
>>>>>>>>>>
>>>>>>>>>> Nirmal Fernando.
>>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>>
>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Imesh Gunaratne
>>>>>>>>>
>>>>>>>>> Technical Lead, WSO2
>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Raj
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Gayan Gunarathne
>>>>>>> Technical Lead
>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Nirmal
>>>>>>
>>>>>> Nirmal Fernando.
>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>
>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Raj
>>>>
>>>
>>>
>>>
>>> --
>>> Raj
>>>
>>
>>
>>
>> --
>> Akila Ravihansa Perera
>> Software Engineer, WSO2
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Akila Ravihansa Perera
Software Engineer, WSO2
Blog: http://ravihansa3000.blogspot.com
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
On Thu, Nov 6, 2014 at 8:33 AM, Akila Ravihansa Perera <ra...@wso2.com>
wrote:
> Thanks for the explanation, Nirmal. It makes sense now.
>
> Perhaps as an alternative solution, window processor can be set to wait
> until sufficient events have arrived to calculate the gradient?
>
What would be the sufficient value? Theoretically in gradient case it's 2,
in second derivative case it's 3. And we're doing that already.
>
> On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> In other words, fix reduces the risks which are coming from bad samples.
>>
>> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>>> > wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> No Imesh. There are no difference between *taking the the difference
>>>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>>>> difference*.
>>>>>
>>>>
>>>> Raj: I agree, in this scenario it makes no difference, however when we
>>>> are applying a fomular the unit conversion should happen prior to applying
>>>> values.
>>>>
>>>
>>> Not exactly Imesh. We can do operations if operands are in the same
>>> units. So we can do subtract operation in milliseconds. And then we can do
>>> division operation to make it in seconds. On the other hand, we can do a
>>> division operation to make each operands to seconds, then since both are
>>> seconds, we can do a subtract operation. So we can apply the formula if
>>> operands are in same unit. I guess this is same for all scenarios.
>>>
>>>
>>>> Still I see a problem with this correction, take the same example:
>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>
>>>> *Manual calulation:*
>>>> The value difference: 7 - 12 = -5
>>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 = 0.005
>>>> sec
>>>> *Gradient = - 5/0.005 = -1000*
>>>>
>>>> *According to the current fix:*
>>>> long millisecondsForASecond = 1000;
>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>> millisecondsForASecond;
>>>> double gradient = 0.0;
>>>> if (tGap > 0) {
>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond)
>>>> / tGap;
>>>> }
>>>>
>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>> Gradient = -5 * 1000 / 1000 = - 5
>>>>
>>>
>>> IMO, this is a sufficient solution.
>>>
>>> Let me explain how I see Nirmal's solution.
>>>
>>> We are taking only first and last values in a window for gradient
>>> calculation. If there are more events in that window, and if that is a good
>>> sample, tGap will be taking some higher values than 1s. This is usual
>>> situation. Nirmal's fix is nothing to do with this scenario.
>>>
>>> If there are few events in that window(say only 2), they can be really
>>> nearer. So there is a possibility that tGap can take a value near to 0. In
>>> that case gradient will be a very big value. Hence, if the tGap is very low
>>> (less than 1 second), then we can assume that the first and last events
>>> actually occurred in 1 second gap. The reason why tGap is less than 0 is,
>>> there are only few events in that window, and that sample is not good
>>> enough. But this a rare situation, it can only happen once in a while.
>>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>>> in the time interval between two events. This fix is really important to
>>> deal with samples which are not good enough.
>>>
>>> By previous formula, we get -1000. This happened because the sample is
>>> not good enough (may be there are few events). Fix makes it to -5.
>>>
>>> So my understanding is that the solution is good enough for now.
>>>
>>> Thanks.
>>>
>>>>
>>>> Nirmal: Can you please justify this?
>>>>
>>>> Thanks
>>>>
>>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <
>>>> nirmal070125@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> AFAIU this is not related with the gradient calculation formula that
>>>>>> we used. But It is related with the how we calculate gradient when the
>>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>>> approaches to the small number we do need to normalize the result.
>>>>>>
>>>>>> IMO we do have max and min scale up and down values. So based on that
>>>>>> we can calculate the gradient on that value range.Simply if the denominator
>>>>>> approaches zero result approaches to infinity.
>>>>>>
>>>>>> I think that's what Nirmal doing here by making the lower threshold
>>>>>> as "millisecondsForASecond"
>>>>>>
>>>>>
>>>>> You got it! thanks.
>>>>>
>>>>>>
>>>>>>
>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>> millisecondsForASecond;
>>>>>> double gradient = 0.0;
>>>>>> if (tGap > 0) {
>>>>>> gradient = ((lastVal - firstVal) *
>>>>>> millisecondsForASecond) / tGap;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Gayan
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Nirmal,
>>>>>>>>
>>>>>>>> I do not think the milliseconds to seconds convertion is correct
>>>>>>>> here.
>>>>>>>>
>>>>>>>> As I see we are taking the difference of two timestamp values and
>>>>>>>> then dividing it by 1000. The corect way might be to first divide each
>>>>>>>> value by 1000 and then take the difference.
>>>>>>>>
>>>>>>>
>>>>>>> No Imesh. There are no difference between *taking the the
>>>>>>> difference and dividing it by 1000* and *dividing it by 1000 and
>>>>>>> taking the difference*.
>>>>>>>
>>>>>>>
>>>>>>>> On the other hand we might not need to convert these values to
>>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>>
>>>>>>>> I did a quick test with the following sample:
>>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>>
>>>>>>>> *According previous code:*
>>>>>>>> long tGap = t2 - t1;
>>>>>>>> double gradient = 0.0;
>>>>>>>> if (tGap > 0) {
>>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>>> }
>>>>>>>>
>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>> gradient: -1000.0
>>>>>>>>
>>>>>>>> *According to your fix:*
>>>>>>>> long millisecondsForASecond = 1000;
>>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>>> millisecondsForASecond;
>>>>>>>> double gradient = 0.0;
>>>>>>>> if (tGap > 0) {
>>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>>> millisecondsForASecond) / tGap;
>>>>>>>> }
>>>>>>>>
>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>> gradient: -5.0
>>>>>>>>
>>>>>>>> *According to an online gradient calculator:*
>>>>>>>> gradient: -1
>>>>>>>>
>>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>>
>>>>>>>> According to the online gradient calculator (assuming their
>>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>>> believe the logic should be simple as follows:
>>>>>>>>
>>>>>>>> long tGap = t2 - t1;
>>>>>>>> double gradient = 0.0;
>>>>>>>> if (tGap > 0) {
>>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>>> }
>>>>>>>>
>>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>>> gradient: -1.0
>>>>>>>>
>>>>>>>
>>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>>> (v1-v2)/timeInSeconds.
>>>>>>>
>>>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>>>> result's unit will be different.
>>>>>>>
>>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your case,
>>>>>>> you will get *stats/milliseconds*
>>>>>>>
>>>>>>> And AS prediction is based on *seconds*, CEP should use *seconds*
>>>>>>> for the calculation.
>>>>>>>
>>>>>>> wdyt?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Guys,
>>>>>>>>>
>>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>>
>>>>>>>>> How we calculate gradient of two events?
>>>>>>>>>
>>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>>
>>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>>
>>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>>
>>>>>>>>> Hence,
>>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 -
>>>>>>>>> t1)*
>>>>>>>>>
>>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>>
>>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>>
>>>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>>>
>>>>>>>>> ===================================================================
>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>>>>
>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>>>>
>>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>>> ===================================================================
>>>>>>>>>
>>>>>>>>> So, as you can see the reason behind a large value is when the
>>>>>>>>> time gap between two subjected events is less than 1 second. This could
>>>>>>>>> happen since events are coming from different asynchronous agents and also
>>>>>>>>> when there are less number of events.
>>>>>>>>>
>>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>>> compromise anything AFAIS.
>>>>>>>>>
>>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>>> *____ t2
>>>>>>>>> - t1*
>>>>>>>>> *| yes?*
>>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>>> *|____
>>>>>>>>> 1000*
>>>>>>>>>
>>>>>>>>> I have tested this and works fine.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>>>>>> derivative of load average
>>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Raj,
>>>>>>>>>
>>>>>>>>> Yes it has. But it's very easy to find the values from the agent
>>>>>>>>> with the time stamp. Then we can search that time stamp in the CEP trace
>>>>>>>>> logs and compare the summarized values.
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>
>>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent
>>>>>>>>>> also right?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika
>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go though
>>>>>>>>>>>>> the logs I'm unable to find the high input values for load average. But
>>>>>>>>>>>>> output have high values.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>>
>>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This is happening intermittently. Experienced now too. This is
>>>>>>>>>>>> a serious issue. We need to somehow sort this out.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also networkPartitionId is empty. In the current execution
>>>>>>>>>>>>> plans it took networkPartitionId for the calculation. IMO we need to update
>>>>>>>>>>>>> the CEP execution plans for 4.1.0.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika
>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>>>>>> previous mail.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see
>>>>>>>>>>>>>>>>> what are the load average values sent from the agent. If those values are
>>>>>>>>>>>>>>>>> normal then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And
>>>>>>>>>>>>>>>>>>> faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers
>>>>>>>>>>>>>>>>>>> are destroyed.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Nirmal
>>>>>>>>>
>>>>>>>>> Nirmal Fernando.
>>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>>
>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Imesh Gunaratne
>>>>>>>>
>>>>>>>> Technical Lead, WSO2
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Raj
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Gayan Gunarathne
>>>>>> Technical Lead
>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nirmal
>>>>>
>>>>> Nirmal Fernando.
>>>>> PPMC Member & Committer of Apache Stratos,
>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>> Raj
>>>
>>
>>
>>
>> --
>> Raj
>>
>
>
>
> --
> Akila Ravihansa Perera
> Software Engineer, WSO2
>
> Blog: http://ravihansa3000.blogspot.com
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Thanks for the explanation, Nirmal. It makes sense now.
Perhaps as an alternative solution, window processor can be set to wait
until sufficient events have arrived to calculate the gradient?
On Wed, Nov 5, 2014 at 11:48 PM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> In other words, fix reduces the risks which are coming from bad samples.
>
> On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>>
>>
>> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> No Imesh. There are no difference between *taking the the difference
>>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>>> difference*.
>>>>
>>>
>>> Raj: I agree, in this scenario it makes no difference, however when we
>>> are applying a fomular the unit conversion should happen prior to applying
>>> values.
>>>
>>
>> Not exactly Imesh. We can do operations if operands are in the same
>> units. So we can do subtract operation in milliseconds. And then we can do
>> division operation to make it in seconds. On the other hand, we can do a
>> division operation to make each operands to seconds, then since both are
>> seconds, we can do a subtract operation. So we can apply the formula if
>> operands are in same unit. I guess this is same for all scenarios.
>>
>>
>>> Still I see a problem with this correction, take the same example:
>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
>>> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>
>>> *Manual calulation:*
>>> The value difference: 7 - 12 = -5
>>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 = 0.005
>>> sec
>>> *Gradient = - 5/0.005 = -1000*
>>>
>>> *According to the current fix:*
>>> long millisecondsForASecond = 1000;
>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>> millisecondsForASecond;
>>> double gradient = 0.0;
>>> if (tGap > 0) {
>>> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
>>> tGap;
>>> }
>>>
>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>> Gradient = -5 * 1000 / 1000 = - 5
>>>
>>
>> IMO, this is a sufficient solution.
>>
>> Let me explain how I see Nirmal's solution.
>>
>> We are taking only first and last values in a window for gradient
>> calculation. If there are more events in that window, and if that is a good
>> sample, tGap will be taking some higher values than 1s. This is usual
>> situation. Nirmal's fix is nothing to do with this scenario.
>>
>> If there are few events in that window(say only 2), they can be really
>> nearer. So there is a possibility that tGap can take a value near to 0. In
>> that case gradient will be a very big value. Hence, if the tGap is very low
>> (less than 1 second), then we can assume that the first and last events
>> actually occurred in 1 second gap. The reason why tGap is less than 0 is,
>> there are only few events in that window, and that sample is not good
>> enough. But this a rare situation, it can only happen once in a while.
>> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
>> in the time interval between two events. This fix is really important to
>> deal with samples which are not good enough.
>>
>> By previous formula, we get -1000. This happened because the sample is
>> not good enough (may be there are few events). Fix makes it to -5.
>>
>> So my understanding is that the solution is good enough for now.
>>
>> Thanks.
>>
>>>
>>> Nirmal: Can you please justify this?
>>>
>>> Thanks
>>>
>>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <nirmal070125@gmail.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>>> wrote:
>>>>
>>>>> AFAIU this is not related with the gradient calculation formula that
>>>>> we used. But It is related with the how we calculate gradient when the
>>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>>> approaches to the small number we do need to normalize the result.
>>>>>
>>>>> IMO we do have max and min scale up and down values. So based on that
>>>>> we can calculate the gradient on that value range.Simply if the denominator
>>>>> approaches zero result approaches to infinity.
>>>>>
>>>>> I think that's what Nirmal doing here by making the lower threshold as
>>>>> "millisecondsForASecond"
>>>>>
>>>>
>>>> You got it! thanks.
>>>>
>>>>>
>>>>>
>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>> millisecondsForASecond;
>>>>> double gradient = 0.0;
>>>>> if (tGap > 0) {
>>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond)
>>>>> / tGap;
>>>>> }
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Gayan
>>>>>
>>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Nirmal,
>>>>>>>
>>>>>>> I do not think the milliseconds to seconds convertion is correct
>>>>>>> here.
>>>>>>>
>>>>>>> As I see we are taking the difference of two timestamp values and
>>>>>>> then dividing it by 1000. The corect way might be to first divide each
>>>>>>> value by 1000 and then take the difference.
>>>>>>>
>>>>>>
>>>>>> No Imesh. There are no difference between *taking the the difference
>>>>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>>>>> difference*.
>>>>>>
>>>>>>
>>>>>>> On the other hand we might not need to convert these values to
>>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>>
>>>>>>> I did a quick test with the following sample:
>>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>>
>>>>>>> *According previous code:*
>>>>>>> long tGap = t2 - t1;
>>>>>>> double gradient = 0.0;
>>>>>>> if (tGap > 0) {
>>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>>> }
>>>>>>>
>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>> gradient: -1000.0
>>>>>>>
>>>>>>> *According to your fix:*
>>>>>>> long millisecondsForASecond = 1000;
>>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>>> millisecondsForASecond;
>>>>>>> double gradient = 0.0;
>>>>>>> if (tGap > 0) {
>>>>>>> gradient = ((lastVal - firstVal) *
>>>>>>> millisecondsForASecond) / tGap;
>>>>>>> }
>>>>>>>
>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>> gradient: -5.0
>>>>>>>
>>>>>>> *According to an online gradient calculator:*
>>>>>>> gradient: -1
>>>>>>>
>>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>>
>>>>>>> According to the online gradient calculator (assuming their
>>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>>> believe the logic should be simple as follows:
>>>>>>>
>>>>>>> long tGap = t2 - t1;
>>>>>>> double gradient = 0.0;
>>>>>>> if (tGap > 0) {
>>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>>> }
>>>>>>>
>>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>>> gradient: -1.0
>>>>>>>
>>>>>>
>>>>>> And AFAIK, standard way to calculate gradient =
>>>>>> (v1-v2)/timeInSeconds.
>>>>>>
>>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>>> result's unit will be different.
>>>>>>
>>>>>> In Nirmal's case, he get will get *stats/seconds*. In your case, you
>>>>>> will get *stats/milliseconds*
>>>>>>
>>>>>> And AS prediction is based on *seconds*, CEP should use *seconds*
>>>>>> for the calculation.
>>>>>>
>>>>>> wdyt?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Guys,
>>>>>>>>
>>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>>
>>>>>>>> How we calculate gradient of two events?
>>>>>>>>
>>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>>
>>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>>
>>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>>
>>>>>>>> Hence,
>>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>>>>>>>
>>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>>
>>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>>
>>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>>
>>>>>>>> ===================================================================
>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>>>
>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>>>
>>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>>> 1415213232154 hash: 155426542
>>>>>>>> ===================================================================
>>>>>>>>
>>>>>>>> So, as you can see the reason behind a large value is when the time
>>>>>>>> gap between two subjected events is less than 1 second. This could happen
>>>>>>>> since events are coming from different asynchronous agents and also when
>>>>>>>> there are less number of events.
>>>>>>>>
>>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>>> compromise anything AFAIS.
>>>>>>>>
>>>>>>>> Fix is to calculate time gap as follows;
>>>>>>>> *____ t2 -
>>>>>>>> t1*
>>>>>>>> *| yes?*
>>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>>> *|____
>>>>>>>> 1000*
>>>>>>>>
>>>>>>>> I have tested this and works fine.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------- Forwarded message ----------
>>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>>>>> derivative of load average
>>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Raj,
>>>>>>>>
>>>>>>>> Yes it has. But it's very easy to find the values from the agent
>>>>>>>> with the time stamp. Then we can search that time stamp in the CEP trace
>>>>>>>> logs and compare the summarized values.
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>>
>>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raj,
>>>>>>>>>>
>>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>>> down where the actual problem is.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent also
>>>>>>>>> right?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte
>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>
>>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go though
>>>>>>>>>>>> the logs I'm unable to find the high input values for load average. But
>>>>>>>>>>>> output have high values.
>>>>>>>>>>>>
>>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>>
>>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is happening intermittently. Experienced now too. This is a
>>>>>>>>>>> serious issue. We need to somehow sort this out.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Also networkPartitionId is empty. In the current execution
>>>>>>>>>>>> plans it took networkPartitionId for the calculation. IMO we need to update
>>>>>>>>>>>> the CEP execution plans for 4.1.0.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte
>>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>>>>> previous mail.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what
>>>>>>>>>>>>>>>> are the load average values sent from the agent. If those values are normal
>>>>>>>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And
>>>>>>>>>>>>>>>>>> faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers
>>>>>>>>>>>>>>>>>> are destroyed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>
>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>> phone : +94 772492511
>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Nirmal
>>>>>>>>
>>>>>>>> Nirmal Fernando.
>>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>>
>>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Imesh Gunaratne
>>>>>>>
>>>>>>> Technical Lead, WSO2
>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Raj
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Gayan Gunarathne
>>>>> Technical Lead
>>>>> WSO2 Inc. (http://wso2.com)
>>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Nirmal
>>>>
>>>> Nirmal Fernando.
>>>> PPMC Member & Committer of Apache Stratos,
>>>> Senior Software Engineer, WSO2 Inc.
>>>>
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>> Raj
>>
>
>
>
> --
> Raj
>
--
Akila Ravihansa Perera
Software Engineer, WSO2
Blog: http://ravihansa3000.blogspot.com
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
In other words, fix reduces the risks which are coming from bad samples.
On Thu, Nov 6, 2014 at 12:08 PM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
>
>
> On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>> Hi,
>>>
>>> No Imesh. There are no difference between *taking the the difference
>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>> difference*.
>>>
>>
>> Raj: I agree, in this scenario it makes no difference, however when we
>> are applying a fomular the unit conversion should happen prior to applying
>> values.
>>
>
> Not exactly Imesh. We can do operations if operands are in the same units.
> So we can do subtract operation in milliseconds. And then we can do
> division operation to make it in seconds. On the other hand, we can do a
> division operation to make each operands to seconds, then since both are
> seconds, we can do a subtract operation. So we can apply the formula if
> operands are in same unit. I guess this is same for all scenarios.
>
>
>> Still I see a problem with this correction, take the same example:
>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
>> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>
>> *Manual calulation:*
>> The value difference: 7 - 12 = -5
>> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 = 0.005
>> sec
>> *Gradient = - 5/0.005 = -1000*
>>
>> *According to the current fix:*
>> long millisecondsForASecond = 1000;
>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>> millisecondsForASecond;
>> double gradient = 0.0;
>> if (tGap > 0) {
>> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
>> tGap;
>> }
>>
>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>> Gradient = -5 * 1000 / 1000 = - 5
>>
>
> IMO, this is a sufficient solution.
>
> Let me explain how I see Nirmal's solution.
>
> We are taking only first and last values in a window for gradient
> calculation. If there are more events in that window, and if that is a good
> sample, tGap will be taking some higher values than 1s. This is usual
> situation. Nirmal's fix is nothing to do with this scenario.
>
> If there are few events in that window(say only 2), they can be really
> nearer. So there is a possibility that tGap can take a value near to 0. In
> that case gradient will be a very big value. Hence, if the tGap is very low
> (less than 1 second), then we can assume that the first and last events
> actually occurred in 1 second gap. The reason why tGap is less than 0 is,
> there are only few events in that window, and that sample is not good
> enough. But this a rare situation, it can only happen once in a while.
> Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
> in the time interval between two events. This fix is really important to
> deal with samples which are not good enough.
>
> By previous formula, we get -1000. This happened because the sample is not
> good enough (may be there are few events). Fix makes it to -5.
>
> So my understanding is that the solution is good enough for now.
>
> Thanks.
>
>>
>> Nirmal: Can you please justify this?
>>
>> Thanks
>>
>> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <ni...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com>
>>> wrote:
>>>
>>>> AFAIU this is not related with the gradient calculation formula that we
>>>> used. But It is related with the how we calculate gradient when the
>>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>>> approaches to the small number we do need to normalize the result.
>>>>
>>>> IMO we do have max and min scale up and down values. So based on that
>>>> we can calculate the gradient on that value range.Simply if the denominator
>>>> approaches zero result approaches to infinity.
>>>>
>>>> I think that's what Nirmal doing here by making the lower threshold as
>>>> "millisecondsForASecond"
>>>>
>>>
>>> You got it! thanks.
>>>
>>>>
>>>>
>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>> millisecondsForASecond;
>>>> double gradient = 0.0;
>>>> if (tGap > 0) {
>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond)
>>>> / tGap;
>>>> }
>>>>
>>>>
>>>> Thanks,
>>>> Gayan
>>>>
>>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>>> > wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Nirmal,
>>>>>>
>>>>>> I do not think the milliseconds to seconds convertion is correct
>>>>>> here.
>>>>>>
>>>>>> As I see we are taking the difference of two timestamp values and
>>>>>> then dividing it by 1000. The corect way might be to first divide each
>>>>>> value by 1000 and then take the difference.
>>>>>>
>>>>>
>>>>> No Imesh. There are no difference between *taking the the difference
>>>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>>>> difference*.
>>>>>
>>>>>
>>>>>> On the other hand we might not need to convert these values to
>>>>>> seconds since we are taking a time difference and calculating a gradient.
>>>>>>
>>>>>> I did a quick test with the following sample:
>>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>>
>>>>>> *According previous code:*
>>>>>> long tGap = t2 - t1;
>>>>>> double gradient = 0.0;
>>>>>> if (tGap > 0) {
>>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>>> }
>>>>>>
>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>> gradient: -1000.0
>>>>>>
>>>>>> *According to your fix:*
>>>>>> long millisecondsForASecond = 1000;
>>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>>> millisecondsForASecond;
>>>>>> double gradient = 0.0;
>>>>>> if (tGap > 0) {
>>>>>> gradient = ((lastVal - firstVal) *
>>>>>> millisecondsForASecond) / tGap;
>>>>>> }
>>>>>>
>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>> gradient: -5.0
>>>>>>
>>>>>> *According to an online gradient calculator:*
>>>>>> gradient: -1
>>>>>>
>>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>>
>>>>>> According to the online gradient calculator (assuming their
>>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>>> believe the logic should be simple as follows:
>>>>>>
>>>>>> long tGap = t2 - t1;
>>>>>> double gradient = 0.0;
>>>>>> if (tGap > 0) {
>>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>>> }
>>>>>>
>>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>>> gradient: -1.0
>>>>>>
>>>>>
>>>>> And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
>>>>>
>>>>> Well, we can divide it by what ever time units we wants, but the
>>>>> result's unit will be different.
>>>>>
>>>>> In Nirmal's case, he get will get *stats/seconds*. In your case, you
>>>>> will get *stats/milliseconds*
>>>>>
>>>>> And AS prediction is based on *seconds*, CEP should use *seconds* for
>>>>> the calculation.
>>>>>
>>>>> wdyt?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>>> nirmal070125@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Guys,
>>>>>>>
>>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>>
>>>>>>> How we calculate gradient of two events?
>>>>>>>
>>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>>
>>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>>
>>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>>
>>>>>>> Hence,
>>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>>>>>>
>>>>>>> I've enabled debug logs for CEP extension;
>>>>>>>
>>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>>
>>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>>
>>>>>>> ===================================================================
>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>>
>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>>
>>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>>> 1415213232154 hash: 155426542
>>>>>>> ===================================================================
>>>>>>>
>>>>>>> So, as you can see the reason behind a large value is when the time
>>>>>>> gap between two subjected events is less than 1 second. This could happen
>>>>>>> since events are coming from different asynchronous agents and also when
>>>>>>> there are less number of events.
>>>>>>>
>>>>>>> So, the fix I propose is a very simple one and it will not
>>>>>>> compromise anything AFAIS.
>>>>>>>
>>>>>>> Fix is to calculate time gap as follows;
>>>>>>> *____ t2 -
>>>>>>> t1*
>>>>>>> *| yes?*
>>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>>> *|____ 1000*
>>>>>>>
>>>>>>> I have tested this and works fine.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------- Forwarded message ----------
>>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>>>> derivative of load average
>>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>>
>>>>>>>
>>>>>>> Hi Raj,
>>>>>>>
>>>>>>> Yes it has. But it's very easy to find the values from the agent
>>>>>>> with the time stamp. Then we can search that time stamp in the CEP trace
>>>>>>> logs and compare the summarized values.
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Raj,
>>>>>>>>>
>>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can
>>>>>>>>> compare both CEP and agent logs with the time stamp. Then we can narrow
>>>>>>>>> down where the actual problem is.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Manula, cep-trace log have the values sent by cartridge agent also
>>>>>>>> right?
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte
>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>
>>>>>>>>>>> Is this happen frequently or intermittently ? Once I go though
>>>>>>>>>>> the logs I'm unable to find the high input values for load average. But
>>>>>>>>>>> output have high values.
>>>>>>>>>>>
>>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>>
>>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is happening intermittently. Experienced now too. This is a
>>>>>>>>>> serious issue. We need to somehow sort this out.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Also networkPartitionId is empty. In the current execution plans
>>>>>>>>>>> it took networkPartitionId for the calculation. IMO we need to update the
>>>>>>>>>>> CEP execution plans for 4.1.0.
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte
>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>>>> previous mail.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what
>>>>>>>>>>>>>>> are the load average values sent from the agent. If those values are normal
>>>>>>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time
>>>>>>>>>>>>>>>> period? It will have the events published from the agent.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And
>>>>>>>>>>>>>>>>> faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers
>>>>>>>>>>>>>>>>> are destroyed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>
>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Nirmal
>>>>>>>
>>>>>>> Nirmal Fernando.
>>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>>
>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Raj
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Gayan Gunarathne
>>>> Technical Lead
>>>> WSO2 Inc. (http://wso2.com)
>>>> email : gayang@wso2.com | mobile : +94 766819985
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> Raj
>
--
Raj
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org> wrote:
> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Hi,
>>
>> No Imesh. There are no difference between *taking the the difference and
>> dividing it by 1000* and *dividing it by 1000 and taking the difference*
>> .
>>
>
> Raj: I agree, in this scenario it makes no difference, however when we are
> applying a fomular the unit conversion should happen prior to applying
> values.
>
Not exactly Imesh. We can do operations if operands are in the same units.
So we can do subtract operation in milliseconds. And then we can do
division operation to make it in seconds. On the other hand, we can do a
division operation to make each operands to seconds, then since both are
seconds, we can do a subtract operation. So we can apply the formula if
operands are in same unit. I guess this is same for all scenarios.
> Still I see a problem with this correction, take the same example:
> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>
> *Manual calulation:*
> The value difference: 7 - 12 = -5
> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 = 0.005 sec
> *Gradient = - 5/0.005 = -1000*
>
> *According to the current fix:*
> long millisecondsForASecond = 1000;
> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
> millisecondsForASecond;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
> tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> Gradient = -5 * 1000 / 1000 = - 5
>
IMO, this is a sufficient solution.
Let me explain how I see Nirmal's solution.
We are taking only first and last values in a window for gradient
calculation. If there are more events in that window, and if that is a good
sample, tGap will be taking some higher values than 1s. This is usual
situation. Nirmal's fix is nothing to do with this scenario.
If there are few events in that window(say only 2), they can be really
nearer. So there is a possibility that tGap can take a value near to 0. In
that case gradient will be a very big value. Hence, if the tGap is very low
(less than 1 second), then we can assume that the first and last events
actually occurred in 1 second gap. The reason why tGap is less than 0 is,
there are only few events in that window, and that sample is not good
enough. But this a rare situation, it can only happen once in a while.
Nirmal's fix is only dealing with this scenario. Fix imposes a lower limit
in the time interval between two events. This fix is really important to
deal with samples which are not good enough.
By previous formula, we get -1000. This happened because the sample is not
good enough (may be there are few events). Fix makes it to -5.
So my understanding is that the solution is good enough for now.
Thanks.
>
> Nirmal: Can you please justify this?
>
> Thanks
>
> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
>>
>>
>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com> wrote:
>>
>>> AFAIU this is not related with the gradient calculation formula that we
>>> used. But It is related with the how we calculate gradient when the
>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>> approaches to the small number we do need to normalize the result.
>>>
>>> IMO we do have max and min scale up and down values. So based on that we
>>> can calculate the gradient on that value range.Simply if the denominator
>>> approaches zero result approaches to infinity.
>>>
>>> I think that's what Nirmal doing here by making the lower threshold as
>>> "millisecondsForASecond"
>>>
>>
>> You got it! thanks.
>>
>>>
>>>
>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>> millisecondsForASecond;
>>> double gradient = 0.0;
>>> if (tGap > 0) {
>>> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
>>> tGap;
>>> }
>>>
>>>
>>> Thanks,
>>> Gayan
>>>
>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Nirmal,
>>>>>
>>>>> I do not think the milliseconds to seconds convertion is correct here.
>>>>>
>>>>> As I see we are taking the difference of two timestamp values and then
>>>>> dividing it by 1000. The corect way might be to first divide each value by
>>>>> 1000 and then take the difference.
>>>>>
>>>>
>>>> No Imesh. There are no difference between *taking the the difference
>>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>>> difference*.
>>>>
>>>>
>>>>> On the other hand we might not need to convert these values to seconds
>>>>> since we are taking a time difference and calculating a gradient.
>>>>>
>>>>> I did a quick test with the following sample:
>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>
>>>>> *According previous code:*
>>>>> long tGap = t2 - t1;
>>>>> double gradient = 0.0;
>>>>> if (tGap > 0) {
>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>> }
>>>>>
>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>> gradient: -1000.0
>>>>>
>>>>> *According to your fix:*
>>>>> long millisecondsForASecond = 1000;
>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>> millisecondsForASecond;
>>>>> double gradient = 0.0;
>>>>> if (tGap > 0) {
>>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond)
>>>>> / tGap;
>>>>> }
>>>>>
>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>> gradient: -5.0
>>>>>
>>>>> *According to an online gradient calculator:*
>>>>> gradient: -1
>>>>>
>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>
>>>>> According to the online gradient calculator (assuming their
>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>> believe the logic should be simple as follows:
>>>>>
>>>>> long tGap = t2 - t1;
>>>>> double gradient = 0.0;
>>>>> if (tGap > 0) {
>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>> }
>>>>>
>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>> gradient: -1.0
>>>>>
>>>>
>>>> And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
>>>>
>>>> Well, we can divide it by what ever time units we wants, but the
>>>> result's unit will be different.
>>>>
>>>> In Nirmal's case, he get will get *stats/seconds*. In your case, you
>>>> will get *stats/milliseconds*
>>>>
>>>> And AS prediction is based on *seconds*, CEP should use *seconds* for
>>>> the calculation.
>>>>
>>>> wdyt?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>> nirmal070125@gmail.com> wrote:
>>>>>
>>>>>> Hi Guys,
>>>>>>
>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>
>>>>>> How we calculate gradient of two events?
>>>>>>
>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>
>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>
>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>
>>>>>> Hence,
>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>>>>>
>>>>>> I've enabled debug logs for CEP extension;
>>>>>>
>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>
>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>
>>>>>> ===================================================================
>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>
>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>
>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>> 1415213232154 hash: 155426542
>>>>>> ===================================================================
>>>>>>
>>>>>> So, as you can see the reason behind a large value is when the time
>>>>>> gap between two subjected events is less than 1 second. This could happen
>>>>>> since events are coming from different asynchronous agents and also when
>>>>>> there are less number of events.
>>>>>>
>>>>>> So, the fix I propose is a very simple one and it will not compromise
>>>>>> anything AFAIS.
>>>>>>
>>>>>> Fix is to calculate time gap as follows;
>>>>>> *____ t2 -
>>>>>> t1*
>>>>>> *| yes?*
>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>> *|____ 1000*
>>>>>>
>>>>>> I have tested this and works fine.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>>> derivative of load average
>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>
>>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> Yes it has. But it's very easy to find the values from the agent with
>>>>>> the time stamp. Then we can search that time stamp in the CEP trace logs
>>>>>> and compare the summarized values.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>>>>> manulac@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Raj,
>>>>>>>>
>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>>>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>>>>>> the actual problem is.
>>>>>>>>
>>>>>>>
>>>>>>> Manula, cep-trace log have the values sent by cartridge agent also
>>>>>>> right?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>>
>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raj,
>>>>>>>>>>
>>>>>>>>>> Is this happen frequently or intermittently ? Once I go though
>>>>>>>>>> the logs I'm unable to find the high input values for load average. But
>>>>>>>>>> output have high values.
>>>>>>>>>>
>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>
>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is happening intermittently. Experienced now too. This is a
>>>>>>>>> serious issue. We need to somehow sort this out.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Also networkPartitionId is empty. In the current execution plans
>>>>>>>>>> it took networkPartitionId for the calculation. IMO we need to update the
>>>>>>>>>> CEP execution plans for 4.1.0.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>
>>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>>> previous mail.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1.
>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what
>>>>>>>>>>>>>> are the load average values sent from the agent. If those values are normal
>>>>>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period?
>>>>>>>>>>>>>>> It will have the events published from the agent.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And
>>>>>>>>>>>>>>>> faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>> phone : +94 772492511
>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Nirmal
>>>>>>
>>>>>> Nirmal Fernando.
>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>
>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Raj
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Gayan Gunarathne
>>> Technical Lead
>>> WSO2 Inc. (http://wso2.com)
>>> email : gayang@wso2.com | mobile : +94 766819985
>>>
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Raj
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Isuru Haththotuwa <is...@apache.org>.
Thanks a lot Nirmal for the detailed explanation. It makes sense for me, as
an effort to decrease the risk of false alarms, such as triggering scale
up, due to input data (in this case, events) not being sufficient to make
a correct decision.
On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne <im...@apache.org> wrote:
> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Hi,
>>
>> No Imesh. There are no difference between *taking the the difference and
>> dividing it by 1000* and *dividing it by 1000 and taking the difference*
>> .
>>
>
> Raj: I agree, in this scenario it makes no difference, however when we are
> applying a fomular the unit conversion should happen prior to applying
> values.
>
> Still I see a problem with this correction, take the same example:
> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>
> *Manual calulation:*
> The value difference: 7 - 12 = -5
> Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 = 0.005 sec
> *Gradient = - 5/0.005 = -1000*
>
> *According to the current fix:*
> long millisecondsForASecond = 1000;
> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
> millisecondsForASecond;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
> tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> Gradient = -5 * 1000 / 1000 = - 5
>
> Nirmal: Can you please justify this?
>
> Thanks
>
AFAIU, we cannot compare these two calculations. The manual calculation is
the standard way of calculating the gradient, which would give correct
answers based on how correct the input values are. It can also give false
values, when in short term there seem to be a couple spikes spanning across
a small time gap, but if you take a larger time frame, there is no much
difference. That it why you are getting the gradient as a large value in
the manual calculation.
AFAIU Nirmal's fix attempts to address this problem. In the real scenario,
we do not want the system to scale up if there are two spikes very close
together, but in the larger time window the values are normal. AFAIU, this
is what the proposed fix covers. Of course, it might not be a perfect
solution, but AFAIU this is will do for now.
>
> On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
>>
>>
>> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com> wrote:
>>
>>> AFAIU this is not related with the gradient calculation formula that we
>>> used. But It is related with the how we calculate gradient when the
>>> denominator become a small number.IMO when the divisor (timeInSeconds)
>>> approaches to the small number we do need to normalize the result.
>>>
>>> IMO we do have max and min scale up and down values. So based on that we
>>> can calculate the gradient on that value range.Simply if the denominator
>>> approaches zero result approaches to infinity.
>>>
>>> I think that's what Nirmal doing here by making the lower threshold as
>>> "millisecondsForASecond"
>>>
>>
>> You got it! thanks.
>>
>>>
>>>
>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>> millisecondsForASecond;
>>> double gradient = 0.0;
>>> if (tGap > 0) {
>>> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
>>> tGap;
>>> }
>>>
>>>
>>> Thanks,
>>> Gayan
>>>
>>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Nirmal,
>>>>>
>>>>> I do not think the milliseconds to seconds convertion is correct here.
>>>>>
>>>>> As I see we are taking the difference of two timestamp values and then
>>>>> dividing it by 1000. The corect way might be to first divide each value by
>>>>> 1000 and then take the difference.
>>>>>
>>>>
>>>> No Imesh. There are no difference between *taking the the difference
>>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>>> difference*.
>>>>
>>>>
>>>>> On the other hand we might not need to convert these values to seconds
>>>>> since we are taking a time difference and calculating a gradient.
>>>>>
>>>>> I did a quick test with the following sample:
>>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>>
>>>>> *According previous code:*
>>>>> long tGap = t2 - t1;
>>>>> double gradient = 0.0;
>>>>> if (tGap > 0) {
>>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>>> }
>>>>>
>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>> gradient: -1000.0
>>>>>
>>>>> *According to your fix:*
>>>>> long millisecondsForASecond = 1000;
>>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>>> millisecondsForASecond;
>>>>> double gradient = 0.0;
>>>>> if (tGap > 0) {
>>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond)
>>>>> / tGap;
>>>>> }
>>>>>
>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>> gradient: -5.0
>>>>>
>>>>> *According to an online gradient calculator:*
>>>>> gradient: -1
>>>>>
>>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>>
>>>>> According to the online gradient calculator (assuming their
>>>>> calculation is correct), the calculation in your fix is not correct. I
>>>>> believe the logic should be simple as follows:
>>>>>
>>>>> long tGap = t2 - t1;
>>>>> double gradient = 0.0;
>>>>> if (tGap > 0) {
>>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>>> }
>>>>>
>>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>>> gradient: -1.0
>>>>>
>>>>
>>>> And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
>>>>
>>>> Well, we can divide it by what ever time units we wants, but the
>>>> result's unit will be different.
>>>>
>>>> In Nirmal's case, he get will get *stats/seconds*. In your case, you
>>>> will get *stats/milliseconds*
>>>>
>>>> And AS prediction is based on *seconds*, CEP should use *seconds* for
>>>> the calculation.
>>>>
>>>> wdyt?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <
>>>>> nirmal070125@gmail.com> wrote:
>>>>>
>>>>>> Hi Guys,
>>>>>>
>>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>>
>>>>>> How we calculate gradient of two events?
>>>>>>
>>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>>
>>>>>> tx - time in *milliseconds* when xth event occurred
>>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>>
>>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>>
>>>>>> Hence,
>>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>>>>>
>>>>>> I've enabled debug logs for CEP extension;
>>>>>>
>>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>>
>>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>>
>>>>>> ===================================================================
>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>>
>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>>
>>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>>> 1415213232154 hash: 155426542
>>>>>> ===================================================================
>>>>>>
>>>>>> So, as you can see the reason behind a large value is when the time
>>>>>> gap between two subjected events is less than 1 second. This could happen
>>>>>> since events are coming from different asynchronous agents and also when
>>>>>> there are less number of events.
>>>>>>
>>>>>> So, the fix I propose is a very simple one and it will not compromise
>>>>>> anything AFAIS.
>>>>>>
>>>>>> Fix is to calculate time gap as follows;
>>>>>> *____ t2 -
>>>>>> t1*
>>>>>> *| yes?*
>>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>>> *|____ 1000*
>>>>>>
>>>>>> I have tested this and works fine.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>>> derivative of load average
>>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>>
>>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> Yes it has. But it's very easy to find the values from the agent with
>>>>>> the time stamp. Then we can search that time stamp in the CEP trace logs
>>>>>> and compare the summarized values.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>>>>> manulac@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Raj,
>>>>>>>>
>>>>>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>>>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>>>>>> the actual problem is.
>>>>>>>>
>>>>>>>
>>>>>>> Manula, cep-trace log have the values sent by cartridge agent also
>>>>>>> right?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>>
>>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raj,
>>>>>>>>>>
>>>>>>>>>> Is this happen frequently or intermittently ? Once I go though
>>>>>>>>>> the logs I'm unable to find the high input values for load average. But
>>>>>>>>>> output have high values.
>>>>>>>>>>
>>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>>
>>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is happening intermittently. Experienced now too. This is a
>>>>>>>>> serious issue. We need to somehow sort this out.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Also networkPartitionId is empty. In the current execution plans
>>>>>>>>>> it took networkPartitionId for the calculation. IMO we need to update the
>>>>>>>>>> CEP execution plans for 4.1.0.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>>
>>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>>> previous mail.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1.
>>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika
>>>>>>>>>>>>> Thantriwatte <ma...@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what
>>>>>>>>>>>>>> are the load average values sent from the agent. If those values are normal
>>>>>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period?
>>>>>>>>>>>>>>> It will have the events published from the agent.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And
>>>>>>>>>>>>>>>> faced this very strange scenario, $Subject.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>> phone : +94 772492511
>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Nirmal
>>>>>>
>>>>>> Nirmal Fernando.
>>>>>> PPMC Member & Committer of Apache Stratos,
>>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>>
>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Raj
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Gayan Gunarathne
>>> Technical Lead
>>> WSO2 Inc. (http://wso2.com)
>>> email : gayang@wso2.com | mobile : +94 766819985
>>>
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Imesh Gunaratne <im...@apache.org>.
On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> Hi,
>
> No Imesh. There are no difference between *taking the the difference and
> dividing it by 1000* and *dividing it by 1000 and taking the difference*.
>
Raj: I agree, in this scenario it makes no difference, however when we are
applying a fomular the unit conversion should happen prior to applying
values.
Still I see a problem with this correction, take the same example:
Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
Time Gap: 5 t1: 1415213232152 t2: 1415213232157
*Manual calulation:*
The value difference: 7 - 12 = -5
Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 = 0.005 sec
*Gradient = - 5/0.005 = -1000*
*According to the current fix:*
long millisecondsForASecond = 1000;
long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
millisecondsForASecond;
double gradient = 0.0;
if (tGap > 0) {
gradient = ((lastVal - firstVal) * millisecondsForASecond) /
tGap;
}
t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
Gradient = -5 * 1000 / 1000 = - 5
Nirmal: Can you please justify this?
Thanks
On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando <ni...@gmail.com>
wrote:
>
>
> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com> wrote:
>
>> AFAIU this is not related with the gradient calculation formula that we
>> used. But It is related with the how we calculate gradient when the
>> denominator become a small number.IMO when the divisor (timeInSeconds)
>> approaches to the small number we do need to normalize the result.
>>
>> IMO we do have max and min scale up and down values. So based on that we
>> can calculate the gradient on that value range.Simply if the denominator
>> approaches zero result approaches to infinity.
>>
>> I think that's what Nirmal doing here by making the lower threshold as
>> "millisecondsForASecond"
>>
>
> You got it! thanks.
>
>>
>>
>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>> millisecondsForASecond;
>> double gradient = 0.0;
>> if (tGap > 0) {
>> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
>> tGap;
>> }
>>
>>
>> Thanks,
>> Gayan
>>
>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>> Hi,
>>>
>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> Hi Nirmal,
>>>>
>>>> I do not think the milliseconds to seconds convertion is correct here.
>>>>
>>>> As I see we are taking the difference of two timestamp values and then
>>>> dividing it by 1000. The corect way might be to first divide each value by
>>>> 1000 and then take the difference.
>>>>
>>>
>>> No Imesh. There are no difference between *taking the the difference
>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>> difference*.
>>>
>>>
>>>> On the other hand we might not need to convert these values to seconds
>>>> since we are taking a time difference and calculating a gradient.
>>>>
>>>> I did a quick test with the following sample:
>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>
>>>> *According previous code:*
>>>> long tGap = t2 - t1;
>>>> double gradient = 0.0;
>>>> if (tGap > 0) {
>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>> }
>>>>
>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>> gradient: -1000.0
>>>>
>>>> *According to your fix:*
>>>> long millisecondsForASecond = 1000;
>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>> millisecondsForASecond;
>>>> double gradient = 0.0;
>>>> if (tGap > 0) {
>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond)
>>>> / tGap;
>>>> }
>>>>
>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>> gradient: -5.0
>>>>
>>>> *According to an online gradient calculator:*
>>>> gradient: -1
>>>>
>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>
>>>> According to the online gradient calculator (assuming their calculation
>>>> is correct), the calculation in your fix is not correct. I believe the
>>>> logic should be simple as follows:
>>>>
>>>> long tGap = t2 - t1;
>>>> double gradient = 0.0;
>>>> if (tGap > 0) {
>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>> }
>>>>
>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>> gradient: -1.0
>>>>
>>>
>>> And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
>>>
>>> Well, we can divide it by what ever time units we wants, but the
>>> result's unit will be different.
>>>
>>> In Nirmal's case, he get will get *stats/seconds*. In your case, you
>>> will get *stats/milliseconds*
>>>
>>> And AS prediction is based on *seconds*, CEP should use *seconds* for
>>> the calculation.
>>>
>>> wdyt?
>>>
>>> Thanks.
>>>
>>>
>>>> Thanks
>>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <nirmal070125@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Guys,
>>>>>
>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>
>>>>> How we calculate gradient of two events?
>>>>>
>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>
>>>>> tx - time in *milliseconds* when xth event occurred
>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>
>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>
>>>>> Hence,
>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>>>>
>>>>> I've enabled debug logs for CEP extension;
>>>>>
>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>
>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>
>>>>> ===================================================================
>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>
>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>
>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>> 1415213232154 hash: 155426542
>>>>> ===================================================================
>>>>>
>>>>> So, as you can see the reason behind a large value is when the time
>>>>> gap between two subjected events is less than 1 second. This could happen
>>>>> since events are coming from different asynchronous agents and also when
>>>>> there are less number of events.
>>>>>
>>>>> So, the fix I propose is a very simple one and it will not compromise
>>>>> anything AFAIS.
>>>>>
>>>>> Fix is to calculate time gap as follows;
>>>>> *____ t2 - t1*
>>>>> *| yes?*
>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>> *|____ 1000*
>>>>>
>>>>> I have tested this and works fine.
>>>>>
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>> derivative of load average
>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>
>>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Yes it has. But it's very easy to find the values from the agent with
>>>>> the time stamp. Then we can search that time stamp in the CEP trace logs
>>>>> and compare the summarized values.
>>>>>
>>>>> Thanks !
>>>>>
>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>>>> manulac@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Raj,
>>>>>>>
>>>>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>>>>> the actual problem is.
>>>>>>>
>>>>>>
>>>>>> Manula, cep-trace log have the values sent by cartridge agent also
>>>>>> right?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Raj,
>>>>>>>>>
>>>>>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>>>>>> logs I'm unable to find the high input values for load average. But output
>>>>>>>>> have high values.
>>>>>>>>>
>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>
>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is happening intermittently. Experienced now too. This is a
>>>>>>>> serious issue. We need to somehow sort this out.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also networkPartitionId is empty. In the current execution plans
>>>>>>>>> it took networkPartitionId for the calculation. IMO we need to update the
>>>>>>>>> CEP execution plans for 4.1.0.
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>
>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>> previous mail.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>
>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>
>>>>>>>>>>>> 1.
>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte
>>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what
>>>>>>>>>>>>> are the load average values sent from the agent. If those values are normal
>>>>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period?
>>>>>>>>>>>>>> It will have the events published from the agent.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rajkumar Rajaratnam
>>>>>> Software Engineer | WSO2, Inc.
>>>>>> Mobile +94777568639 | +94783498120
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nirmal
>>>>>
>>>>> Nirmal Fernando.
>>>>> PPMC Member & Committer of Apache Stratos,
>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>> Raj
>>>
>>
>>
>>
>> --
>>
>> Gayan Gunarathne
>> Technical Lead
>> WSO2 Inc. (http://wso2.com)
>> email : gayang@wso2.com | mobile : +94 766819985
>>
>>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
As you can see, currently it's the linear gradient that we're calculating,
IMO which is sufficient enough estimation.
One could improve this if they wish. One such an improvement would be to
find a median of events and use it as the second event. But I wonder
whether, that's necessary.
On Thu, Nov 6, 2014 at 7:01 AM, Nirmal Fernando <ni...@gmail.com>
wrote:
>
>
> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com> wrote:
>
>> AFAIU this is not related with the gradient calculation formula that we
>> used. But It is related with the how we calculate gradient when the
>> denominator become a small number.IMO when the divisor (timeInSeconds)
>> approaches to the small number we do need to normalize the result.
>>
>> IMO we do have max and min scale up and down values. So based on that we
>> can calculate the gradient on that value range.Simply if the denominator
>> approaches zero result approaches to infinity.
>>
>> I think that's what Nirmal doing here by making the lower threshold as
>> "millisecondsForASecond"
>>
>
> You got it! thanks.
>
>>
>>
>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>> millisecondsForASecond;
>> double gradient = 0.0;
>> if (tGap > 0) {
>> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
>> tGap;
>> }
>>
>>
>> Thanks,
>> Gayan
>>
>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>> Hi,
>>>
>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> Hi Nirmal,
>>>>
>>>> I do not think the milliseconds to seconds convertion is correct here.
>>>>
>>>> As I see we are taking the difference of two timestamp values and then
>>>> dividing it by 1000. The corect way might be to first divide each value by
>>>> 1000 and then take the difference.
>>>>
>>>
>>> No Imesh. There are no difference between *taking the the difference
>>> and dividing it by 1000* and *dividing it by 1000 and taking the
>>> difference*.
>>>
>>>
>>>> On the other hand we might not need to convert these values to seconds
>>>> since we are taking a time difference and calculating a gradient.
>>>>
>>>> I did a quick test with the following sample:
>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val:
>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>>
>>>> *According previous code:*
>>>> long tGap = t2 - t1;
>>>> double gradient = 0.0;
>>>> if (tGap > 0) {
>>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>>> }
>>>>
>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>> gradient: -1000.0
>>>>
>>>> *According to your fix:*
>>>> long millisecondsForASecond = 1000;
>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>>> millisecondsForASecond;
>>>> double gradient = 0.0;
>>>> if (tGap > 0) {
>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond)
>>>> / tGap;
>>>> }
>>>>
>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>> gradient: -5.0
>>>>
>>>> *According to an online gradient calculator:*
>>>> gradient: -1
>>>>
>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>>
>>>> According to the online gradient calculator (assuming their calculation
>>>> is correct), the calculation in your fix is not correct. I believe the
>>>> logic should be simple as follows:
>>>>
>>>> long tGap = t2 - t1;
>>>> double gradient = 0.0;
>>>> if (tGap > 0) {
>>>> gradient = ((lastVal - firstVal)) / tGap;
>>>> }
>>>>
>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>>> gradient: -1.0
>>>>
>>>
>>> And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
>>>
>>> Well, we can divide it by what ever time units we wants, but the
>>> result's unit will be different.
>>>
>>> In Nirmal's case, he get will get *stats/seconds*. In your case, you
>>> will get *stats/milliseconds*
>>>
>>> And AS prediction is based on *seconds*, CEP should use *seconds* for
>>> the calculation.
>>>
>>> wdyt?
>>>
>>> Thanks.
>>>
>>>
>>>> Thanks
>>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <nirmal070125@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Guys,
>>>>>
>>>>> So, I got a chance to reproduce and analyze this issue.
>>>>>
>>>>> How we calculate gradient of two events?
>>>>>
>>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>>
>>>>> tx - time in *milliseconds* when xth event occurred
>>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>>
>>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>>
>>>>> Hence,
>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>>>>
>>>>> I've enabled debug logs for CEP extension;
>>>>>
>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>>
>>>>> Please find the following 3 logs extracted from the debug logs;
>>>>>
>>>>> ===================================================================
>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>>
>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>>
>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>>> 1415213232154 hash: 155426542
>>>>> ===================================================================
>>>>>
>>>>> So, as you can see the reason behind a large value is when the time
>>>>> gap between two subjected events is less than 1 second. This could happen
>>>>> since events are coming from different asynchronous agents and also when
>>>>> there are less number of events.
>>>>>
>>>>> So, the fix I propose is a very simple one and it will not compromise
>>>>> anything AFAIS.
>>>>>
>>>>> Fix is to calculate time gap as follows;
>>>>> *____ t2 - t1*
>>>>> *| yes?*
>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>>> *|____ 1000*
>>>>>
>>>>> I have tested this and works fine.
>>>>>
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>>> Subject: Re: CEP sends very large values for gradient and second
>>>>> derivative of load average
>>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>>
>>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Yes it has. But it's very easy to find the values from the agent with
>>>>> the time stamp. Then we can search that time stamp in the CEP trace logs
>>>>> and compare the summarized values.
>>>>>
>>>>> Thanks !
>>>>>
>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>>>> manulac@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Raj,
>>>>>>>
>>>>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>>>>> the actual problem is.
>>>>>>>
>>>>>>
>>>>>> Manula, cep-trace log have the values sent by cartridge agent also
>>>>>> right?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Raj,
>>>>>>>>>
>>>>>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>>>>>> logs I'm unable to find the high input values for load average. But output
>>>>>>>>> have high values.
>>>>>>>>>
>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>>
>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is happening intermittently. Experienced now too. This is a
>>>>>>>> serious issue. We need to somehow sort this out.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also networkPartitionId is empty. In the current execution plans
>>>>>>>>> it took networkPartitionId for the calculation. IMO we need to update the
>>>>>>>>> CEP execution plans for 4.1.0.
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Manula,
>>>>>>>>>>>
>>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>>> previous mail.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>>
>>>>>>>>>>>> I couldn't find any large values sent by agent. There are
>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>>
>>>>>>>>>>>> 1.
>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte
>>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what
>>>>>>>>>>>>> are the load average values sent from the agent. If those values are normal
>>>>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period?
>>>>>>>>>>>>>> It will have the events published from the agent.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rajkumar Rajaratnam
>>>>>> Software Engineer | WSO2, Inc.
>>>>>> Mobile +94777568639 | +94783498120
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Nirmal
>>>>>
>>>>> Nirmal Fernando.
>>>>> PPMC Member & Committer of Apache Stratos,
>>>>> Senior Software Engineer, WSO2 Inc.
>>>>>
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>> Raj
>>>
>>
>>
>>
>> --
>>
>> Gayan Gunarathne
>> Technical Lead
>> WSO2 Inc. (http://wso2.com)
>> email : gayang@wso2.com | mobile : +94 766819985
>>
>>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Nirmal Fernando <ni...@gmail.com>.
On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne <ga...@wso2.com> wrote:
> AFAIU this is not related with the gradient calculation formula that we
> used. But It is related with the how we calculate gradient when the
> denominator become a small number.IMO when the divisor (timeInSeconds)
> approaches to the small number we do need to normalize the result.
>
> IMO we do have max and min scale up and down values. So based on that we
> can calculate the gradient on that value range.Simply if the denominator
> approaches zero result approaches to infinity.
>
> I think that's what Nirmal doing here by making the lower threshold as
> "millisecondsForASecond"
>
You got it! thanks.
>
>
> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
> millisecondsForASecond;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
> tGap;
> }
>
>
> Thanks,
> Gayan
>
> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Hi,
>>
>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org> wrote:
>>
>>> Hi Nirmal,
>>>
>>> I do not think the milliseconds to seconds convertion is correct here.
>>>
>>> As I see we are taking the difference of two timestamp values and then
>>> dividing it by 1000. The corect way might be to first divide each value by
>>> 1000 and then take the difference.
>>>
>>
>> No Imesh. There are no difference between *taking the the difference and
>> dividing it by 1000* and *dividing it by 1000 and taking the difference*.
>>
>>
>>
>>> On the other hand we might not need to convert these values to seconds
>>> since we are taking a time difference and calculating a gradient.
>>>
>>> I did a quick test with the following sample:
>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
>>> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>>
>>> *According previous code:*
>>> long tGap = t2 - t1;
>>> double gradient = 0.0;
>>> if (tGap > 0) {
>>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>>> }
>>>
>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>> gradient: -1000.0
>>>
>>> *According to your fix:*
>>> long millisecondsForASecond = 1000;
>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>>> millisecondsForASecond;
>>> double gradient = 0.0;
>>> if (tGap > 0) {
>>> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
>>> tGap;
>>> }
>>>
>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>> gradient: -5.0
>>>
>>> *According to an online gradient calculator:*
>>> gradient: -1
>>>
>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>>
>>> According to the online gradient calculator (assuming their calculation
>>> is correct), the calculation in your fix is not correct. I believe the
>>> logic should be simple as follows:
>>>
>>> long tGap = t2 - t1;
>>> double gradient = 0.0;
>>> if (tGap > 0) {
>>> gradient = ((lastVal - firstVal)) / tGap;
>>> }
>>>
>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>>> gradient: -1.0
>>>
>>
>> And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
>>
>> Well, we can divide it by what ever time units we wants, but the result's
>> unit will be different.
>>
>> In Nirmal's case, he get will get *stats/seconds*. In your case, you
>> will get *stats/milliseconds*
>>
>> And AS prediction is based on *seconds*, CEP should use *seconds* for
>> the calculation.
>>
>> wdyt?
>>
>> Thanks.
>>
>>
>>> Thanks
>>>
>>>
>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <ni...@gmail.com>
>>> wrote:
>>>
>>>> Hi Guys,
>>>>
>>>> So, I got a chance to reproduce and analyze this issue.
>>>>
>>>> How we calculate gradient of two events?
>>>>
>>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>>
>>>> tx - time in *milliseconds* when xth event occurred
>>>> vx - value (memory, cpu etc.) that xth event carries
>>>>
>>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>>
>>>> Hence,
>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>>>
>>>> I've enabled debug logs for CEP extension;
>>>>
>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>>
>>>> Please find the following 3 logs extracted from the debug logs;
>>>>
>>>> ===================================================================
>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time
>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>>
>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>>
>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>>> 1415213232154 hash: 155426542
>>>> ===================================================================
>>>>
>>>> So, as you can see the reason behind a large value is when the time gap
>>>> between two subjected events is less than 1 second. This could happen since
>>>> events are coming from different asynchronous agents and also when there
>>>> are less number of events.
>>>>
>>>> So, the fix I propose is a very simple one and it will not compromise
>>>> anything AFAIS.
>>>>
>>>> Fix is to calculate time gap as follows;
>>>> *____ t2 - t1*
>>>> *| yes?*
>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>>> *|____ 1000*
>>>>
>>>> I have tested this and works fine.
>>>>
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>>> Subject: Re: CEP sends very large values for gradient and second
>>>> derivative of load average
>>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>>
>>>>
>>>> Hi Raj,
>>>>
>>>> Yes it has. But it's very easy to find the values from the agent with
>>>> the time stamp. Then we can search that time stamp in the CEP trace logs
>>>> and compare the summarized values.
>>>>
>>>> Thanks !
>>>>
>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>>> manulac@wso2.com> wrote:
>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>>>> the actual problem is.
>>>>>>
>>>>>
>>>>> Manula, cep-trace log have the values sent by cartridge agent also
>>>>> right?
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>>>> manulac@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Raj,
>>>>>>>>
>>>>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>>>>> logs I'm unable to find the high input values for load average. But output
>>>>>>>> have high values.
>>>>>>>>
>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>>> JMSOutputAdaptor, sent
>>>>>>>>
>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>>
>>>>>>>
>>>>>>> This is happening intermittently. Experienced now too. This is a
>>>>>>> serious issue. We need to somehow sort this out.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>>>>>> execution plans for 4.1.0.
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Manula,
>>>>>>>>>>
>>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>>> previous mail.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Chamila,
>>>>>>>>>>>
>>>>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>>
>>>>>>>>>>> 1.
>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte
>>>>>>>>>>> <ma...@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>>>>> the load average values sent from the agent. If those values are normal
>>>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks !
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period?
>>>>>>>>>>>>> It will have the events published from the agent.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>
>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>> phone : +94 772492511
>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Software Engineer | WSO2, Inc.
>>>>> Mobile +94777568639 | +94783498120
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manulac@wso2.com / manula@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Nirmal
>>>>
>>>> Nirmal Fernando.
>>>> PPMC Member & Committer of Apache Stratos,
>>>> Senior Software Engineer, WSO2 Inc.
>>>>
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>> Raj
>>
>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email : gayang@wso2.com | mobile : +94 766819985
>
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Gayan Gunarathne <ga...@wso2.com>.
AFAIU this is not related with the gradient calculation formula that we
used. But It is related with the how we calculate gradient when the
denominator become a small number.IMO when the divisor (timeInSeconds)
approaches to the small number we do need to normalize the result.
IMO we do have max and min scale up and down values. So based on that we
can calculate the gradient on that value range.Simply if the denominator
approaches zero result approaches to infinity.
I think that's what Nirmal doing here by making the lower threshold as
"millisecondsForASecond"
long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
millisecondsForASecond;
double gradient = 0.0;
if (tGap > 0) {
gradient = ((lastVal - firstVal) * millisecondsForASecond) /
tGap;
}
Thanks,
Gayan
On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> Hi,
>
> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Hi Nirmal,
>>
>> I do not think the milliseconds to seconds convertion is correct here.
>>
>> As I see we are taking the difference of two timestamp values and then
>> dividing it by 1000. The corect way might be to first divide each value by
>> 1000 and then take the difference.
>>
>
> No Imesh. There are no difference between *taking the the difference and
> dividing it by 1000* and *dividing it by 1000 and taking the difference*.
>
>
>> On the other hand we might not need to convert these values to seconds
>> since we are taking a time difference and calculating a gradient.
>>
>> I did a quick test with the following sample:
>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
>> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>>
>> *According previous code:*
>> long tGap = t2 - t1;
>> double gradient = 0.0;
>> if (tGap > 0) {
>> gradient = ((lastVal - firstVal) * 1000) / tGap;
>> }
>>
>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>> gradient: -1000.0
>>
>> *According to your fix:*
>> long millisecondsForASecond = 1000;
>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
>> millisecondsForASecond;
>> double gradient = 0.0;
>> if (tGap > 0) {
>> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
>> tGap;
>> }
>>
>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>> gradient: -5.0
>>
>> *According to an online gradient calculator:*
>> gradient: -1
>>
>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>>
>> According to the online gradient calculator (assuming their calculation
>> is correct), the calculation in your fix is not correct. I believe the
>> logic should be simple as follows:
>>
>> long tGap = t2 - t1;
>> double gradient = 0.0;
>> if (tGap > 0) {
>> gradient = ((lastVal - firstVal)) / tGap;
>> }
>>
>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
>> gradient: -1.0
>>
>
> And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
>
> Well, we can divide it by what ever time units we wants, but the result's
> unit will be different.
>
> In Nirmal's case, he get will get *stats/seconds*. In your case, you will
> get *stats/milliseconds*
>
> And AS prediction is based on *seconds*, CEP should use *seconds* for the
> calculation.
>
> wdyt?
>
> Thanks.
>
>
>> Thanks
>>
>>
>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <ni...@gmail.com>
>> wrote:
>>
>>> Hi Guys,
>>>
>>> So, I got a chance to reproduce and analyze this issue.
>>>
>>> How we calculate gradient of two events?
>>>
>>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>>
>>> tx - time in *milliseconds* when xth event occurred
>>> vx - value (memory, cpu etc.) that xth event carries
>>>
>>> *time gap = t(2-1) = t2 -t1 milliseconds*
>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>>
>>> Hence,
>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>>
>>> I've enabled debug logs for CEP extension;
>>>
>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>>
>>> Please find the following 3 logs extracted from the debug logs;
>>>
>>> ===================================================================
>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
>>> 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>>
>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>>
>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>>> 1415213232154 hash: 155426542
>>> ===================================================================
>>>
>>> So, as you can see the reason behind a large value is when the time gap
>>> between two subjected events is less than 1 second. This could happen since
>>> events are coming from different asynchronous agents and also when there
>>> are less number of events.
>>>
>>> So, the fix I propose is a very simple one and it will not compromise
>>> anything AFAIS.
>>>
>>> Fix is to calculate time gap as follows;
>>> *____ t2 - t1*
>>> *| yes?*
>>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>>> *|____ 1000*
>>>
>>> I have tested this and works fine.
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>>> Date: Tue, Oct 28, 2014 at 5:16 AM
>>> Subject: Re: CEP sends very large values for gradient and second
>>> derivative of load average
>>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>>
>>>
>>> Hi Raj,
>>>
>>> Yes it has. But it's very easy to find the values from the agent with
>>> the time stamp. Then we can search that time stamp in the CEP trace logs
>>> and compare the summarized values.
>>>
>>> Thanks !
>>>
>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>>> manulac@wso2.com> wrote:
>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>>> the actual problem is.
>>>>>
>>>>
>>>> Manula, cep-trace log have the values sent by cartridge agent also
>>>> right?
>>>>
>>>>
>>>>>
>>>>> Thanks !
>>>>>
>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>>> manulac@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Raj,
>>>>>>>
>>>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>>>> logs I'm unable to find the high input values for load average. But output
>>>>>>> have high values.
>>>>>>>
>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>>> JMSOutputAdaptor, sent
>>>>>>>
>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>>
>>>>>>
>>>>>> This is happening intermittently. Experienced now too. This is a
>>>>>> serious issue. We need to somehow sort this out.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>>>>> execution plans for 4.1.0.
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>>> manulac@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Manula,
>>>>>>>>>
>>>>>>>>> You can see those values in the same log I attached in the
>>>>>>>>> previous mail.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Chamila,
>>>>>>>>>>
>>>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>>
>>>>>>>>>> 1.
>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>>>> the load average values sent from the agent. If those values are normal
>>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>>
>>>>>>>>>>> Thanks !
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>>
>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>>>>> will have the events published from the agent.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>>> 200000.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>>> check executed for :
>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>>
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>> phone : +94 772492511
>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rajkumar Rajaratnam
>>>>>> Software Engineer | WSO2, Inc.
>>>>>> Mobile +94777568639 | +94783498120
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Software Engineer | WSO2, Inc.
>>>> Mobile +94777568639 | +94783498120
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manula Chathurika Thantriwatte
>>> Software Engineer
>>> WSO2 Inc. : http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> email : manulac@wso2.com / manula@apache.org
>>> phone : +94 772492511
>>> blog : http://manulachathurika.blogspot.com/
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> Raj
>
--
Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email : gayang@wso2.com | mobile : +94 766819985
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
Hi,
On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Nirmal,
>
> I do not think the milliseconds to seconds convertion is correct here.
>
> As I see we are taking the difference of two timestamp values and then
> dividing it by 1000. The corect way might be to first divide each value by
> 1000 and then take the difference.
>
No Imesh. There are no difference between *taking the the difference and
dividing it by 1000* and *dividing it by 1000 and taking the difference*.
> On the other hand we might not need to convert these values to seconds
> since we are taking a time difference and calculating a gradient.
>
> I did a quick test with the following sample:
> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
> Time Gap: 5 t1: 1415213232152 t2: 1415213232157
>
> *According previous code:*
> long tGap = t2 - t1;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * 1000) / tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -1000.0
>
> *According to your fix:*
> long millisecondsForASecond = 1000;
> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
> millisecondsForASecond;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal) * millisecondsForASecond) /
> tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -5.0
>
> *According to an online gradient calculator:*
> gradient: -1
>
> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
>
> According to the online gradient calculator (assuming their calculation is
> correct), the calculation in your fix is not correct. I believe the logic
> should be simple as follows:
>
> long tGap = t2 - t1;
> double gradient = 0.0;
> if (tGap > 0) {
> gradient = ((lastVal - firstVal)) / tGap;
> }
>
> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
> gradient: -1.0
>
And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds.
Well, we can divide it by what ever time units we wants, but the result's
unit will be different.
In Nirmal's case, he get will get *stats/seconds*. In your case, you will
get *stats/milliseconds*
And AS prediction is based on *seconds*, CEP should use *seconds* for the
calculation.
wdyt?
Thanks.
> Thanks
>
>
> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
>> Hi Guys,
>>
>> So, I got a chance to reproduce and analyze this issue.
>>
>> How we calculate gradient of two events?
>>
>> Say the events are; e1(t1,v1) and e2(t2,v2)
>>
>> tx - time in *milliseconds* when xth event occurred
>> vx - value (memory, cpu etc.) that xth event carries
>>
>> *time gap = t(2-1) = t2 -t1 milliseconds*
>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>>
>> Hence,
>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>>
>> I've enabled debug logs for CEP extension;
>>
>> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>>
>> Please find the following 3 logs extracted from the debug logs;
>>
>> ===================================================================
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
>> 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>>
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time
>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>>
>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2:
>> 1415213232154 hash: 155426542
>> ===================================================================
>>
>> So, as you can see the reason behind a large value is when the time gap
>> between two subjected events is less than 1 second. This could happen since
>> events are coming from different asynchronous agents and also when there
>> are less number of events.
>>
>> So, the fix I propose is a very simple one and it will not compromise
>> anything AFAIS.
>>
>> Fix is to calculate time gap as follows;
>> *____ t2 - t1*
>> *| yes?*
>> *time gap = t(2-1) = t2 -t1 > 1000 ---*
>> *|____ 1000*
>>
>> I have tested this and works fine.
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
>> Date: Tue, Oct 28, 2014 at 5:16 AM
>> Subject: Re: CEP sends very large values for gradient and second
>> derivative of load average
>> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>>
>>
>> Hi Raj,
>>
>> Yes it has. But it's very easy to find the values from the agent with the
>> time stamp. Then we can search that time stamp in the CEP trace logs and
>> compare the summarized values.
>>
>> Thanks !
>>
>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>>> manulac@wso2.com> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> Shall we enable the cartridge agent debug logs. Then we can compare
>>>> both CEP and agent logs with the time stamp. Then we can narrow down where
>>>> the actual problem is.
>>>>
>>>
>>> Manula, cep-trace log have the values sent by cartridge agent also
>>> right?
>>>
>>>
>>>>
>>>> Thanks !
>>>>
>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>>> manulac@wso2.com> wrote:
>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>>> logs I'm unable to find the high input values for load average. But output
>>>>>> have high values.
>>>>>>
>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>>> JMSOutputAdaptor, sent
>>>>>>
>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>>
>>>>>
>>>>> This is happening intermittently. Experienced now too. This is a
>>>>> serious issue. We need to somehow sort this out.
>>>>>
>>>>>
>>>>>>
>>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>>>> execution plans for 4.1.0.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>>> manulac@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks Raj. I'll go through them.
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Manula,
>>>>>>>>
>>>>>>>> You can see those values in the same log I attached in the previous
>>>>>>>> mail.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Chamila,
>>>>>>>>>
>>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>>
>>>>>>>>> 1.
>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>>> the load average values sent from the agent. If those values are normal
>>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>>
>>>>>>>>>> Thanks !
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raj,
>>>>>>>>>>>
>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>>>> will have the events published from the agent.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Chamila de Alwis
>>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>>> 200000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>>> check executed for :
>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>>>
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>>> intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>>> destroyed.
>>>>>>>>>>>>
>>>>>>>>>>>> Any idea?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Software Engineer | WSO2, Inc.
>>>>> Mobile +94777568639 | +94783498120
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manulac@wso2.com / manula@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos,
>> Senior Software Engineer, WSO2 Inc.
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Raj
Re: [Fixed] CEP sends very large values for gradient and second
derivative of load average
Posted by Imesh Gunaratne <im...@apache.org>.
Hi Nirmal,
I do not think the milliseconds to seconds convertion is correct here.
As I see we are taking the difference of two timestamp values and then
dividing it by 1000. The corect way might be to first divide each value by
1000 and then take the difference.
On the other hand we might not need to convert these values to seconds
since we are taking a time difference and calculating a gradient.
I did a quick test with the following sample:
Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0
Time Gap: 5 t1: 1415213232152 t2: 1415213232157
*According previous code:*
long tGap = t2 - t1;
double gradient = 0.0;
if (tGap > 0) {
gradient = ((lastVal - firstVal) * 1000) / tGap;
}
t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
gradient: -1000.0
*According to your fix:*
long millisecondsForASecond = 1000;
long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 :
millisecondsForASecond;
double gradient = 0.0;
if (tGap > 0) {
gradient = ((lastVal - firstVal) * millisecondsForASecond) /
tGap;
}
t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
gradient: -5.0
*According to an online gradient calculator:*
gradient: -1
http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19
According to the online gradient calculator (assuming their calculation is
correct), the calculation in your fix is not correct. I believe the logic
should be simple as follows:
long tGap = t2 - t1;
double gradient = 0.0;
if (tGap > 0) {
gradient = ((lastVal - firstVal)) / tGap;
}
t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
gradient: -1.0
Thanks
On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando <ni...@gmail.com>
wrote:
> Hi Guys,
>
> So, I got a chance to reproduce and analyze this issue.
>
> How we calculate gradient of two events?
>
> Say the events are; e1(t1,v1) and e2(t2,v2)
>
> tx - time in *milliseconds* when xth event occurred
> vx - value (memory, cpu etc.) that xth event carries
>
> *time gap = t(2-1) = t2 -t1 milliseconds*
> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds
>
> Hence,
> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)*
>
> I've enabled debug logs for CEP extension;
>
> log4j.logger.org.apache.stratos.cep.extension=DEBUG
>
> Please find the following 3 logs extracted from the debug logs;
>
> ===================================================================
> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time Gap:
> 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542
>
> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient:
> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time Gap:
> 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542
>
> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG
> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} -
> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val:
> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2: 1415213232154
> hash: 155426542
> ===================================================================
>
> So, as you can see the reason behind a large value is when the time gap
> between two subjected events is less than 1 second. This could happen since
> events are coming from different asynchronous agents and also when there
> are less number of events.
>
> So, the fix I propose is a very simple one and it will not compromise
> anything AFAIS.
>
> Fix is to calculate time gap as follows;
> *____ t2 - t1*
> *| yes?*
> *time gap = t(2-1) = t2 -t1 > 1000 ---*
> *|____ 1000*
>
> I have tested this and works fine.
>
>
>
> ---------- Forwarded message ----------
> From: Manula Chathurika Thantriwatte <ma...@wso2.com>
> Date: Tue, Oct 28, 2014 at 5:16 AM
> Subject: Re: CEP sends very large values for gradient and second
> derivative of load average
> To: "dev@stratos.apache.org" <de...@stratos.apache.org>
>
>
> Hi Raj,
>
> Yes it has. But it's very easy to find the values from the agent with the
> time stamp. Then we can search that time stamp in the CEP trace logs and
> compare the summarized values.
>
> Thanks !
>
> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>>
>>
>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
>> manulac@wso2.com> wrote:
>>
>>> Hi Raj,
>>>
>>> Shall we enable the cartridge agent debug logs. Then we can compare both
>>> CEP and agent logs with the time stamp. Then we can narrow down where the
>>> actual problem is.
>>>
>>
>> Manula, cep-trace log have the values sent by cartridge agent also right?
>>
>>
>>>
>>> Thanks !
>>>
>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>>> manulac@wso2.com> wrote:
>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Is this happen frequently or intermittently ? Once I go though the
>>>>> logs I'm unable to find the high input values for load average. But output
>>>>> have high values.
>>>>>
>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>>> JMSOutputAdaptor, sent
>>>>>
>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>>
>>>>
>>>> This is happening intermittently. Experienced now too. This is a
>>>> serious issue. We need to somehow sort this out.
>>>>
>>>>
>>>>>
>>>>> Also networkPartitionId is empty. In the current execution plans it
>>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>>> execution plans for 4.1.0.
>>>>>
>>>>> Thanks !
>>>>>
>>>>>
>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>>> manulac@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks Raj. I'll go through them.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Manula,
>>>>>>>
>>>>>>> You can see those values in the same log I attached in the previous
>>>>>>> mail.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Chamila,
>>>>>>>>
>>>>>>>> I couldn't find any large values sent by agent. There are traces
>>>>>>>> for CEP sending large values. Find the cep trace log here [1]
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Shall we enable the cartridge agent debug logs and see what are
>>>>>>>>> the load average values sent from the agent. If those values are normal
>>>>>>>>> then we can narrow down to the CEP.
>>>>>>>>>
>>>>>>>>> Thanks !
>>>>>>>>>
>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raj,
>>>>>>>>>>
>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>>> will have the events published from the agent.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Chamila de Alwis
>>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am doing a test round with M3 with python agent. And faced
>>>>>>>>>>> this very strange scenario, $Subject.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>>> 200000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>>> check executed for :
>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>>> loadAverageReset : true
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>> intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>>> intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>>> destroyed.
>>>>>>>>>>>
>>>>>>>>>>> Any idea?
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Software Engineer | WSO2, Inc.
>>>> Mobile +94777568639 | +94783498120
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manula Chathurika Thantriwatte
>>> Software Engineer
>>> WSO2 Inc. : http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> email : manulac@wso2.com / manula@apache.org
>>> phone : +94 772492511
>>> blog : http://manulachathurika.blogspot.com/
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Software Engineer | WSO2, Inc.
>> Mobile +94777568639 | +94783498120
>>
>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manulac@wso2.com / manula@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
>
>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos