You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Rajkumar Rajaratnam <ra...@wso2.com> on 2014/10/25 19:09:47 UTC
CEP sends very large values for gradient and second derivative of
load average
Hi,
I am doing a test round with M3 with python agent. And faced this very
strange scenario, $Subject.
TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] -2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 1.9090909
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.718182
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of Memory Consumption event: [cluster] php.php.domain [value]
200000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 3.3636363
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 13500.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 50.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.738462
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 2.3333333
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 31.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] php.php.domain [value] -11000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] php.php.domain [value] 28.8
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
KubernetesServiceClusterMonitor is running..
KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
clusterId=php.php.domain, serviceId=php]
TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
[kub-cluster] KubGrp1 [cluster] php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
php.php.domain [Replicas] nonTerminated : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
php.php.domain [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
: php.php.domain [Replicas] obsoleteReplicas : 0
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
check executed for :
org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
flag of rifReset : true flag of memoryConsumptionReset : true flag of
loadAverageReset : true
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
[kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
intervals]: 1
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
intervals]: 1
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
2000000.0 , [time intervals]: 1
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [Replicas] maxReplicas : 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [Replicas] nonTerminated : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [Replicas] activeReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [RequestInFlight] predicted value : 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [RequestInFlight] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [RequestInFlight] lower limit : 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [MemoryConsumption] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [MemoryConsumption] lower limit : 15.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [LoadAverage] predicted value : 989039.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [LoadAverage] upper limit : 180.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [LoadAverage] lower limit : 20.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain scale-up action : true
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain scale-down action : false
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain [LoadAverage] predicted replicas : 16484
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
php.php.domain predicted replicas > max replicas :
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
scale-up : [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
php.php.domain valid number of replicas to expand : 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
{org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
- Updating kubernetes replication controller via cloud controller:
[cluster] php.php.domain [replicas] 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Cloud Controller is delegating request to update a replication controller
php.php.domain to Kubernetes layer.
I couldn't investigate the agent side, since containers are destroyed.
Any idea?
Thanks.
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Manula Chathurika Thantriwatte <ma...@wso2.com>.
Hi Raj,
Yes it has. But it's very easy to find the values from the agent with the
time stamp. Then we can search that time stamp in the CEP trace logs and
compare the summarized values.
Thanks !
On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
>
>
> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
> manulac@wso2.com> wrote:
>
>> Hi Raj,
>>
>> Shall we enable the cartridge agent debug logs. Then we can compare both
>> CEP and agent logs with the time stamp. Then we can narrow down where the
>> actual problem is.
>>
>
> Manula, cep-trace log have the values sent by cartridge agent also right?
>
>
>>
>> Thanks !
>>
>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>>> manulac@wso2.com> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> Is this happen frequently or intermittently ? Once I go though the logs
>>>> I'm unable to find the high input values for load average. But output have
>>>> high values.
>>>>
>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>>> JMSOutputAdaptor, sent
>>>>
>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>>
>>>
>>> This is happening intermittently. Experienced now too. This is a serious
>>> issue. We need to somehow sort this out.
>>>
>>>
>>>>
>>>> Also networkPartitionId is empty. In the current execution plans it
>>>> took networkPartitionId for the calculation. IMO we need to update the CEP
>>>> execution plans for 4.1.0.
>>>>
>>>> Thanks !
>>>>
>>>>
>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>>> manulac@wso2.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thanks Raj. I'll go through them.
>>>>>
>>>>> Thanks !
>>>>>
>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>> Hi Manula,
>>>>>>
>>>>>> You can see those values in the same log I attached in the previous
>>>>>> mail.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Chamila,
>>>>>>>
>>>>>>> I couldn't find any large values sent by agent. There are traces for
>>>>>>> CEP sending large values. Find the cep trace log here [1]
>>>>>>>
>>>>>>> 1.
>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>>> manulac@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Shall we enable the cartridge agent debug logs and see what are the
>>>>>>>> load average values sent from the agent. If those values are normal then we
>>>>>>>> can narrow down to the CEP.
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>>
>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Raj,
>>>>>>>>>
>>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>>> will have the events published from the agent.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Chamila de Alwis
>>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am doing a test round with M3 with python agent. And faced this
>>>>>>>>>> very strange scenario, $Subject.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>>> 200000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>>> check executed for :
>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>>> loadAverageReset : true
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>> intervals]: 1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>>> intervals]: 1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>>> destroyed.
>>>>>>>>>>
>>>>>>>>>> Any idea?
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>> phone : +94 772492511
>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rajkumar Rajaratnam
>>>>>> Software Engineer | WSO2, Inc.
>>>>>> Mobile +94777568639 | +94783498120
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manulac@wso2.com / manula@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>
>
> --
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
> Mobile +94777568639 | +94783498120
>
--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware
email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <
manulac@wso2.com> wrote:
> Hi Raj,
>
> Shall we enable the cartridge agent debug logs. Then we can compare both
> CEP and agent logs with the time stamp. Then we can narrow down where the
> actual problem is.
>
Manula, cep-trace log have the values sent by cartridge agent also right?
>
> Thanks !
>
> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>>
>>
>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
>> manulac@wso2.com> wrote:
>>
>>> Hi Raj,
>>>
>>> Is this happen frequently or intermittently ? Once I go though the logs
>>> I'm unable to find the high input values for load average. But output have
>>> high values.
>>>
>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>>> JMSOutputAdaptor, sent
>>>
>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>>
>>
>> This is happening intermittently. Experienced now too. This is a serious
>> issue. We need to somehow sort this out.
>>
>>
>>>
>>> Also networkPartitionId is empty. In the current execution plans it took
>>> networkPartitionId for the calculation. IMO we need to update the CEP
>>> execution plans for 4.1.0.
>>>
>>> Thanks !
>>>
>>>
>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>>> manulac@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thanks Raj. I'll go through them.
>>>>
>>>> Thanks !
>>>>
>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>> Hi Manula,
>>>>>
>>>>> You can see those values in the same log I attached in the previous
>>>>> mail.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>> Hi Chamila,
>>>>>>
>>>>>> I couldn't find any large values sent by agent. There are traces for
>>>>>> CEP sending large values. Find the cep trace log here [1]
>>>>>>
>>>>>> 1.
>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>>> manulac@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Shall we enable the cartridge agent debug logs and see what are the
>>>>>>> load average values sent from the agent. If those values are normal then we
>>>>>>> can narrow down to the CEP.
>>>>>>>
>>>>>>> Thanks !
>>>>>>>
>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <
>>>>>>> chamilad@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Raj,
>>>>>>>>
>>>>>>>> Can you inspect the CEP trace logs for the same time period? It
>>>>>>>> will have the events published from the agent.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Chamila de Alwis
>>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am doing a test round with M3 with python agent. And faced this
>>>>>>>>> very strange scenario, $Subject.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>>> 200000.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>>> PodActivationWatcher running : Running
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>>> check executed for :
>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>>> loadAverageReset : true
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>> intervals]: 1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>>> intervals]: 1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain scale-up action : true
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain scale-down action : false
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>>> destroyed.
>>>>>>>>>
>>>>>>>>> Any idea?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rajkumar Rajaratnam
>>>>>> Software Engineer | WSO2, Inc.
>>>>>> Mobile +94777568639 | +94783498120
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Software Engineer | WSO2, Inc.
>>>>> Mobile +94777568639 | +94783498120
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manulac@wso2.com / manula@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manula Chathurika Thantriwatte
>>> Software Engineer
>>> WSO2 Inc. : http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> email : manulac@wso2.com / manula@apache.org
>>> phone : +94 772492511
>>> blog : http://manulachathurika.blogspot.com/
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Software Engineer | WSO2, Inc.
>> Mobile +94777568639 | +94783498120
>>
>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manulac@wso2.com / manula@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
>
>
>
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Manula Chathurika Thantriwatte <ma...@wso2.com>.
Hi Raj,
Shall we enable the cartridge agent debug logs. Then we can compare both
CEP and agent logs with the time stamp. Then we can narrow down where the
actual problem is.
Thanks !
On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
>
>
> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
> manulac@wso2.com> wrote:
>
>> Hi Raj,
>>
>> Is this happen frequently or intermittently ? Once I go though the logs
>> I'm unable to find the high input values for load average. But output have
>> high values.
>>
>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
>> JMSOutputAdaptor, sent
>>
>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>>
>
> This is happening intermittently. Experienced now too. This is a serious
> issue. We need to somehow sort this out.
>
>
>>
>> Also networkPartitionId is empty. In the current execution plans it took
>> networkPartitionId for the calculation. IMO we need to update the CEP
>> execution plans for 4.1.0.
>>
>> Thanks !
>>
>>
>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
>> manulac@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Thanks Raj. I'll go through them.
>>>
>>> Thanks !
>>>
>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>> Hi Manula,
>>>>
>>>> You can see those values in the same log I attached in the previous
>>>> mail.
>>>>
>>>> Thanks.
>>>>
>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>> Hi Chamila,
>>>>>
>>>>> I couldn't find any large values sent by agent. There are traces for
>>>>> CEP sending large values. Find the cep trace log here [1]
>>>>>
>>>>> 1.
>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>>> manulac@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Shall we enable the cartridge agent debug logs and see what are the
>>>>>> load average values sent from the agent. If those values are normal then we
>>>>>> can narrow down to the CEP.
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <chamilad@wso2.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi Raj,
>>>>>>>
>>>>>>> Can you inspect the CEP trace logs for the same time period? It will
>>>>>>> have the events published from the agent.
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Chamila de Alwis
>>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>>> Blog: code.chamiladealwis.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am doing a test round with M3 with python agent. And faced this
>>>>>>>> very strange scenario, $Subject.
>>>>>>>>
>>>>>>>>
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>>> 200000.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>> PodActivationWatcher running : Running
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>> PodActivationWatcher running : Running
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>>> PodActivationWatcher running : Running
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>>> check executed for :
>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>>> loadAverageReset : true
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>> intervals]: 1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>>> intervals]: 1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain scale-up action : true
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain scale-down action : false
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>>
>>>>>>>>
>>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>>> destroyed.
>>>>>>>>
>>>>>>>> Any idea?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rajkumar Rajaratnam
>>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Software Engineer | WSO2, Inc.
>>>>> Mobile +94777568639 | +94783498120
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Software Engineer | WSO2, Inc.
>>>> Mobile +94777568639 | +94783498120
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manula Chathurika Thantriwatte
>>> Software Engineer
>>> WSO2 Inc. : http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> email : manulac@wso2.com / manula@apache.org
>>> phone : +94 772492511
>>> blog : http://manulachathurika.blogspot.com/
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>
>
> --
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
> Mobile +94777568639 | +94783498120
>
--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware
email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
manulac@wso2.com> wrote:
> Hi Raj,
>
> Is this happen frequently or intermittently ? Once I go though the logs
> I'm unable to find the high input values for load average. But output have
> high values.
>
> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
> JMSOutputAdaptor, sent
>
> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
>
This is happening intermittently. Experienced now too. This is a serious
issue. We need to somehow sort this out.
>
> Also networkPartitionId is empty. In the current execution plans it took
> networkPartitionId for the calculation. IMO we need to update the CEP
> execution plans for 4.1.0.
>
> Thanks !
>
>
> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
> manulac@wso2.com> wrote:
>
>> Hi,
>>
>> Thanks Raj. I'll go through them.
>>
>> Thanks !
>>
>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>> Hi Manula,
>>>
>>> You can see those values in the same log I attached in the previous mail.
>>>
>>> Thanks.
>>>
>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>> Hi Chamila,
>>>>
>>>> I couldn't find any large values sent by agent. There are traces for
>>>> CEP sending large values. Find the cep trace log here [1]
>>>>
>>>> 1.
>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>>
>>>> Thanks.
>>>>
>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>>> manulac@wso2.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Shall we enable the cartridge agent debug logs and see what are the
>>>>> load average values sent from the agent. If those values are normal then we
>>>>> can narrow down to the CEP.
>>>>>
>>>>> Thanks !
>>>>>
>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <ch...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> Can you inspect the CEP trace logs for the same time period? It will
>>>>>> have the events published from the agent.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Chamila de Alwis
>>>>>> Software Engineer | WSO2 | +94772207163
>>>>>> Blog: code.chamiladealwis.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am doing a test round with M3 with python agent. And faced this
>>>>>>> very strange scenario, $Subject.
>>>>>>>
>>>>>>>
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>>> 200000.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>> PodActivationWatcher running : Running
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>> PodActivationWatcher running : Running
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>>> PodActivationWatcher running : Running
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>>> check executed for :
>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>>> loadAverageReset : true
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>> intervals]: 1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>>> intervals]: 1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>>> 2000000.0 , [time intervals]: 1
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain scale-up action : true
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain scale-down action : false
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>>> scale-up : [cluster] : php.php.domain
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>>> php.php.domain to Kubernetes layer.
>>>>>>>
>>>>>>>
>>>>>>> I couldn't investigate the agent side, since containers are
>>>>>>> destroyed.
>>>>>>>
>>>>>>> Any idea?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> Rajkumar Rajaratnam
>>>>>>> Software Engineer | WSO2, Inc.
>>>>>>> Mobile +94777568639 | +94783498120
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Software Engineer | WSO2, Inc.
>>>> Mobile +94777568639 | +94783498120
>>>>
>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manulac@wso2.com / manula@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
>
>
>
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
On Mon, Oct 27, 2014 at 3:46 PM, Chamila De Alwis <ch...@wso2.com> wrote:
>
> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
> manulac@wso2.com> wrote:
>
>> Also networkPartitionId is empty. In the current execution plans it took
>> networkPartitionId for the calculation. IMO we need to update the CEP
>> execution plans for 4.1.0.
>
>
> +1
>
> As far as I know Kubernetes does not have a network partition Id and a
> partition Id.
>
Yes, we have to create new execution plans for kubernetes. It will publish
stats against cluster id only. Current execution plan publishes stats
against network partition id and cluster id.
Thanks.
>
>
> Regards,
> Chamila de Alwis
> Software Engineer | WSO2 | +94772207163
> Blog: code.chamiladealwis.com
>
>
>
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Chamila De Alwis <ch...@wso2.com>.
On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte <
manulac@wso2.com> wrote:
> Also networkPartitionId is empty. In the current execution plans it took
> networkPartitionId for the calculation. IMO we need to update the CEP
> execution plans for 4.1.0.
+1
As far as I know Kubernetes does not have a network partition Id and a
partition Id.
Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Manula Chathurika Thantriwatte <ma...@wso2.com>.
Hi Raj,
Is this happen frequently or intermittently ? Once I go though the logs I'm
unable to find the high input values for load average. But output have high
values.
22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO
EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor :
JMSOutputAdaptor, sent
{"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}}
Also networkPartitionId is empty. In the current execution plans it took
networkPartitionId for the calculation. IMO we need to update the CEP
execution plans for 4.1.0.
Thanks !
On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte <
manulac@wso2.com> wrote:
> Hi,
>
> Thanks Raj. I'll go through them.
>
> Thanks !
>
> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Hi Manula,
>>
>> You can see those values in the same log I attached in the previous mail.
>>
>> Thanks.
>>
>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <ra...@wso2.com>
>> wrote:
>>
>>> Hi Chamila,
>>>
>>> I couldn't find any large values sent by agent. There are traces for CEP
>>> sending large values. Find the cep trace log here [1]
>>>
>>> 1.
>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>>
>>> Thanks.
>>>
>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>>> manulac@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Shall we enable the cartridge agent debug logs and see what are the
>>>> load average values sent from the agent. If those values are normal then we
>>>> can narrow down to the CEP.
>>>>
>>>> Thanks !
>>>>
>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <ch...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Can you inspect the CEP trace logs for the same time period? It will
>>>>> have the events published from the agent.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Chamila de Alwis
>>>>> Software Engineer | WSO2 | +94772207163
>>>>> Blog: code.chamiladealwis.com
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>>> rajkumarr@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am doing a test round with M3 with python agent. And faced this
>>>>>> very strange scenario, $Subject.
>>>>>>
>>>>>>
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>>> 200000.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>> PodActivationWatcher running : Running
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>> PodActivationWatcher running : Running
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>>> PodActivationWatcher running : Running
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>> KubernetesServiceClusterMonitor is running..
>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>>> clusterId=php.php.domain, serviceId=php]
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>>> check executed for :
>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>>> loadAverageReset : true
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>> intervals]: 1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>>> intervals]: 1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>>> 2000000.0 , [time intervals]: 1
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain scale-up action : true
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain scale-down action : false
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>>> php.php.domain predicted replicas > max replicas :
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>>> scale-up : [cluster] : php.php.domain
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>>> php.php.domain valid number of replicas to expand : 10
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>>> [cluster] php.php.domain [replicas] 10
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>>> Cloud Controller is delegating request to update a replication controller
>>>>>> php.php.domain to Kubernetes layer.
>>>>>>
>>>>>>
>>>>>> I couldn't investigate the agent side, since containers are destroyed.
>>>>>>
>>>>>> Any idea?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> --
>>>>>> Rajkumar Rajaratnam
>>>>>> Software Engineer | WSO2, Inc.
>>>>>> Mobile +94777568639 | +94783498120
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manulac@wso2.com / manula@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Software Engineer | WSO2, Inc.
>> Mobile +94777568639 | +94783498120
>>
>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manulac@wso2.com / manula@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
>
>
>
--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware
email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Manula Chathurika Thantriwatte <ma...@wso2.com>.
Hi,
Thanks Raj. I'll go through them.
Thanks !
On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> Hi Manula,
>
> You can see those values in the same log I attached in the previous mail.
>
> Thanks.
>
> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Hi Chamila,
>>
>> I couldn't find any large values sent by agent. There are traces for CEP
>> sending large values. Find the cep trace log here [1]
>>
>> 1.
>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>>
>> Thanks.
>>
>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
>> manulac@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Shall we enable the cartridge agent debug logs and see what are the load
>>> average values sent from the agent. If those values are normal then we can
>>> narrow down to the CEP.
>>>
>>> Thanks !
>>>
>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <ch...@wso2.com>
>>> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> Can you inspect the CEP trace logs for the same time period? It will
>>>> have the events published from the agent.
>>>>
>>>>
>>>> Regards,
>>>> Chamila de Alwis
>>>> Software Engineer | WSO2 | +94772207163
>>>> Blog: code.chamiladealwis.com
>>>>
>>>>
>>>>
>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>>> rajkumarr@wso2.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am doing a test round with M3 with python agent. And faced this very
>>>>> strange scenario, $Subject.
>>>>>
>>>>>
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>>> 200000.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>> PodActivationWatcher running : Running
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>> PodActivationWatcher running : Running
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>>> PodActivationWatcher running : Running
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>> KubernetesServiceClusterMonitor is running..
>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>>> clusterId=php.php.domain, serviceId=php]
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>>> check executed for :
>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>>> loadAverageReset : true
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>> intervals]: 1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>>> intervals]: 1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>>> 2000000.0 , [time intervals]: 1
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [Replicas] minReplicas : 3
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [Replicas] maxReplicas : 10
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [Replicas] nonTerminated : 3
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [Replicas] activeReplicas : 3
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain scale-up action : true
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain scale-down action : false
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>>> php.php.domain predicted replicas > max replicas :
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>>> scale-up : [cluster] : php.php.domain
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>>> php.php.domain valid number of replicas to expand : 10
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>>> - Updating kubernetes replication controller via cloud controller:
>>>>> [cluster] php.php.domain [replicas] 10
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>>> Cloud Controller is delegating request to update a replication controller
>>>>> php.php.domain to Kubernetes layer.
>>>>>
>>>>>
>>>>> I couldn't investigate the agent side, since containers are destroyed.
>>>>>
>>>>> Any idea?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> --
>>>>> Rajkumar Rajaratnam
>>>>> Software Engineer | WSO2, Inc.
>>>>> Mobile +94777568639 | +94783498120
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manula Chathurika Thantriwatte
>>> Software Engineer
>>> WSO2 Inc. : http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> email : manulac@wso2.com / manula@apache.org
>>> phone : +94 772492511
>>> blog : http://manulachathurika.blogspot.com/
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Software Engineer | WSO2, Inc.
>> Mobile +94777568639 | +94783498120
>>
>
>
>
> --
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
> Mobile +94777568639 | +94783498120
>
--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware
email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
Hi Manula,
You can see those values in the same log I attached in the previous mail.
Thanks.
On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> Hi Chamila,
>
> I couldn't find any large values sent by agent. There are traces for CEP
> sending large values. Find the cep trace log here [1]
>
> 1.
> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
>
> Thanks.
>
> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
> manulac@wso2.com> wrote:
>
>> Hi,
>>
>> Shall we enable the cartridge agent debug logs and see what are the load
>> average values sent from the agent. If those values are normal then we can
>> narrow down to the CEP.
>>
>> Thanks !
>>
>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <ch...@wso2.com>
>> wrote:
>>
>>> Hi Raj,
>>>
>>> Can you inspect the CEP trace logs for the same time period? It will
>>> have the events published from the agent.
>>>
>>>
>>> Regards,
>>> Chamila de Alwis
>>> Software Engineer | WSO2 | +94772207163
>>> Blog: code.chamiladealwis.com
>>>
>>>
>>>
>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <
>>> rajkumarr@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am doing a test round with M3 with python agent. And faced this very
>>>> strange scenario, $Subject.
>>>>
>>>>
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>>> 200000.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> load avg event: [cluster] php.php.domain [value] 5.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> load avg event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> load avg event: [cluster] php.php.domain [value] 31.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> load avg event: [cluster] php.php.domain [value] 39.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>> PodActivationWatcher running : Running
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>> PodActivationWatcher running : Running
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>>> PodActivationWatcher running : Running
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>> KubernetesServiceClusterMonitor is running..
>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>>> clusterId=php.php.domain, serviceId=php]
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>> php.php.domain [Replicas] nonTerminated : 3
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>>> php.php.domain [Replicas] minReplicas : 3
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>>> check executed for :
>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>>> loadAverageReset : true
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>> intervals]: 1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>>> intervals]: 1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>>> 2000000.0 , [time intervals]: 1
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [Replicas] minReplicas : 3
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [Replicas] maxReplicas : 10
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [Replicas] nonTerminated : 3
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [Replicas] activeReplicas : 3
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [LoadAverage] upper limit : 180.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [LoadAverage] lower limit : 20.0
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain scale-up action : true
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain scale-down action : false
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>>> php.php.domain predicted replicas > max replicas :
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>>> scale-up : [cluster] : php.php.domain
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>>> php.php.domain valid number of replicas to expand : 10
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>>> - Updating kubernetes replication controller via cloud controller:
>>>> [cluster] php.php.domain [replicas] 10
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>>> Cloud Controller is delegating request to update a replication controller
>>>> php.php.domain to Kubernetes layer.
>>>>
>>>>
>>>> I couldn't investigate the agent side, since containers are destroyed.
>>>>
>>>> Any idea?
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Software Engineer | WSO2, Inc.
>>>> Mobile +94777568639 | +94783498120
>>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>>
>>
>>
>
>
> --
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
> Mobile +94777568639 | +94783498120
>
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
Hi Chamila,
I couldn't find any large values sent by agent. There are traces for CEP
sending large values. Find the cep trace log here [1]
1.
https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing
Thanks.
On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte <
manulac@wso2.com> wrote:
> Hi,
>
> Shall we enable the cartridge agent debug logs and see what are the load
> average values sent from the agent. If those values are normal then we can
> narrow down to the CEP.
>
> Thanks !
>
> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <ch...@wso2.com>
> wrote:
>
>> Hi Raj,
>>
>> Can you inspect the CEP trace logs for the same time period? It will have
>> the events published from the agent.
>>
>>
>> Regards,
>> Chamila de Alwis
>> Software Engineer | WSO2 | +94772207163
>> Blog: code.chamiladealwis.com
>>
>>
>>
>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>> > wrote:
>>
>>> Hi,
>>>
>>> I am doing a test round with M3 with python agent. And faced this very
>>> strange scenario, $Subject.
>>>
>>>
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> load avg event: [cluster] php.php.domain [value] -2000.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> load avg event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> load avg event: [cluster] php.php.domain [value] 1.9090909
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>>> 200000.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> load avg event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> load avg event: [cluster] php.php.domain [value] 3.3636363
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> load avg event: [cluster] php.php.domain [value] 5.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> load avg event: [cluster] php.php.domain [value] 2000.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> load avg event: [cluster] php.php.domain [value] 13500.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> load avg event: [cluster] php.php.domain [value] 2.3333333
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> load avg event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> load avg event: [cluster] php.php.domain [value] 31.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> load avg event: [cluster] php.php.domain [value] 39.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> load avg event: [cluster] php.php.domain [value] 39.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> load avg event: [cluster] php.php.domain [value] -11000.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>> PodActivationWatcher running : Running
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>> PodActivationWatcher running : Running
>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>>> PodActivationWatcher running : Running
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>> KubernetesServiceClusterMonitor is running..
>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>>> clusterId=php.php.domain, serviceId=php]
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>> php.php.domain [Replicas] nonTerminated : 3
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>>> php.php.domain [Replicas] minReplicas : 3
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>>> : php.php.domain [Replicas] obsoleteReplicas : 0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>>> check executed for :
>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>>> loadAverageReset : true
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>> intervals]: 1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>>> intervals]: 1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>>> 2000000.0 , [time intervals]: 1
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [Replicas] minReplicas : 3
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [Replicas] maxReplicas : 10
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [Replicas] nonTerminated : 3
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [Replicas] activeReplicas : 3
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [RequestInFlight] predicted value : 0.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [RequestInFlight] upper limit : 80.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [RequestInFlight] lower limit : 5.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [MemoryConsumption] upper limit : 80.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [MemoryConsumption] lower limit : 15.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [LoadAverage] predicted value : 989039.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [LoadAverage] upper limit : 180.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [LoadAverage] lower limit : 20.0
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain scale-up action : true
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain scale-down action : false
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain [LoadAverage] predicted replicas : 16484
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>>> php.php.domain predicted replicas > max replicas :
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>>> scale-up : [cluster] : php.php.domain
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>>> php.php.domain valid number of replicas to expand : 10
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>>> - Updating kubernetes replication controller via cloud controller:
>>> [cluster] php.php.domain [replicas] 10
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>>> Cloud Controller is delegating request to update a replication controller
>>> php.php.domain to Kubernetes layer.
>>>
>>>
>>> I couldn't investigate the agent side, since containers are destroyed.
>>>
>>> Any idea?
>>>
>>> Thanks.
>>>
>>> --
>>> Rajkumar Rajaratnam
>>> Software Engineer | WSO2, Inc.
>>> Mobile +94777568639 | +94783498120
>>>
>>
>>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manulac@wso2.com / manula@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
>
>
>
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Manula Chathurika Thantriwatte <ma...@wso2.com>.
Hi,
Shall we enable the cartridge agent debug logs and see what are the load
average values sent from the agent. If those values are normal then we can
narrow down to the CEP.
Thanks !
On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <ch...@wso2.com>
wrote:
> Hi Raj,
>
> Can you inspect the CEP trace logs for the same time period? It will have
> the events published from the agent.
>
>
> Regards,
> Chamila de Alwis
> Software Engineer | WSO2 | +94772207163
> Blog: code.chamiladealwis.com
>
>
>
> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Hi,
>>
>> I am doing a test round with M3 with python agent. And faced this very
>> strange scenario, $Subject.
>>
>>
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] php.php.domain [value] -2000.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] php.php.domain [value] 1.9090909
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
>> 200000.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] php.php.domain [value] 3.3636363
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] php.php.domain [value] 5.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] php.php.domain [value] 2000.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> Memory Consumption event: [cluster] php.php.domain [value] 28.7
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] php.php.domain [value] 13500.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> Memory Consumption event: [cluster] php.php.domain [value] 50.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] php.php.domain [value] 2.3333333
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] php.php.domain [value] 31.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] php.php.domain [value] 39.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] php.php.domain [value] 39.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] php.php.domain [value] -11000.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> Memory Consumption event: [cluster] php.php.domain [value] 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> Memory Consumption event: [cluster] php.php.domain [value] 28.8
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>> KubernetesServiceClusterMonitor is running..
>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>> clusterId=php.php.domain, serviceId=php]
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>> [kub-cluster] KubGrp1 [cluster] php.php.domain
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>> php.php.domain [Replicas] nonTerminated : 3
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>> php.php.domain [Replicas] minReplicas : 3
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>> : php.php.domain [Replicas] obsoleteReplicas : 0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>> check executed for :
>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>> flag of rifReset : true flag of memoryConsumptionReset : true flag of
>> loadAverageReset : true
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>> intervals]: 1
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>> intervals]: 1
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
>> 2000000.0 , [time intervals]: 1
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [Replicas] minReplicas : 3
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [Replicas] maxReplicas : 10
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [Replicas] nonTerminated : 3
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [Replicas] activeReplicas : 3
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [RequestInFlight] predicted value : 0.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [RequestInFlight] upper limit : 80.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [RequestInFlight] lower limit : 5.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [MemoryConsumption] upper limit : 80.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [MemoryConsumption] lower limit : 15.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [LoadAverage] predicted value : 989039.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [LoadAverage] upper limit : 180.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [LoadAverage] lower limit : 20.0
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain scale-up action : true
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain scale-down action : false
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain [LoadAverage] predicted replicas : 16484
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> php.php.domain predicted replicas > max replicas :
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>> scale-up : [cluster] : php.php.domain
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
>> php.php.domain valid number of replicas to expand : 10
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>> - Updating kubernetes replication controller via cloud controller:
>> [cluster] php.php.domain [replicas] 10
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> Cloud Controller is delegating request to update a replication controller
>> php.php.domain to Kubernetes layer.
>>
>>
>> I couldn't investigate the agent side, since containers are destroyed.
>>
>> Any idea?
>>
>> Thanks.
>>
>> --
>> Rajkumar Rajaratnam
>> Software Engineer | WSO2, Inc.
>> Mobile +94777568639 | +94783498120
>>
>
>
--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware
email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Chamila De Alwis <ch...@wso2.com>.
Hi Raj,
Can you inspect the CEP trace logs for the same time period? It will have
the events published from the agent.
Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com
On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> Hi,
>
> I am doing a test round with M3 with python agent. And faced this very
> strange scenario, $Subject.
>
>
> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] -2000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 1.9090909
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
> 200000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 3.3636363
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.7
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.7
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 5.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 2000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.7
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 13500.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 50.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 2.3333333
> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 31.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 39.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 39.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] -11000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.8
> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> KubernetesServiceClusterMonitor is running..
> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
> clusterId=php.php.domain, serviceId=php]
> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
> [kub-cluster] KubGrp1 [cluster] php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
> php.php.domain [Replicas] nonTerminated : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
> php.php.domain [Replicas] minReplicas : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
> : php.php.domain [Replicas] obsoleteReplicas : 0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
> check executed for :
> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> flag of rifReset : true flag of memoryConsumptionReset : true flag of
> loadAverageReset : true
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
> intervals]: 1
> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
> intervals]: 1
> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
> 2000000.0 , [time intervals]: 1
> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [Replicas] minReplicas : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [Replicas] maxReplicas : 10
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [Replicas] nonTerminated : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [Replicas] activeReplicas : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [RequestInFlight] predicted value : 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [RequestInFlight] upper limit : 80.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [RequestInFlight] lower limit : 5.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [MemoryConsumption] upper limit : 80.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [MemoryConsumption] lower limit : 15.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [LoadAverage] predicted value : 989039.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [LoadAverage] upper limit : 180.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [LoadAverage] lower limit : 20.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain scale-up action : true
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain scale-down action : false
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [LoadAverage] predicted replicas : 16484
> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain predicted replicas > max replicas :
> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
> scale-up : [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
> php.php.domain valid number of replicas to expand : 10
> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
> - Updating kubernetes replication controller via cloud controller:
> [cluster] php.php.domain [replicas] 10
> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Cloud Controller is delegating request to update a replication controller
> php.php.domain to Kubernetes layer.
>
>
> I couldn't investigate the agent side, since containers are destroyed.
>
> Any idea?
>
> Thanks.
>
> --
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
> Mobile +94777568639 | +94783498120
>
Re: CEP sends very large values for gradient and second derivative of
load average
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
In the same log, you can see member consumption also has too large value
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.
autoscaler.monitor.KubernetesClusterMonitor} - Second Derivation of Memory
Consumption event: [cluster] php.php.domain [value] 200000.0
On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> Hi,
>
> I am doing a test round with M3 with python agent. And faced this very
> strange scenario, $Subject.
>
>
> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] -2000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 1.9090909
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.718182
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of Memory Consumption event: [cluster] php.php.domain [value]
> 200000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 3.3636363
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.7
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.7
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 5.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 2000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.7
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 13500.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 50.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.738462
> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 2.3333333
> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 31.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 39.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] php.php.domain [value] 39.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] php.php.domain [value] -11000.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] php.php.domain [value] 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] php.php.domain [value] 28.8
> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> KubernetesServiceClusterMonitor is running..
> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
> clusterId=php.php.domain, serviceId=php]
> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
> [kub-cluster] KubGrp1 [cluster] php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
> php.php.domain [Replicas] nonTerminated : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
> php.php.domain [Replicas] minReplicas : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
> : php.php.domain [Replicas] obsoleteReplicas : 0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
> check executed for :
> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> flag of rifReset : true flag of memoryConsumptionReset : true flag of
> loadAverageReset : true
> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
> [kub-cluster] : KubGrp1 [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
> intervals]: 1
> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
> intervals]: 1
> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]:
> 2000000.0 , [time intervals]: 1
> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [Replicas] minReplicas : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [Replicas] maxReplicas : 10
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [Replicas] nonTerminated : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [Replicas] activeReplicas : 3
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [RequestInFlight] predicted value : 0.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [RequestInFlight] upper limit : 80.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [RequestInFlight] lower limit : 5.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [MemoryConsumption] upper limit : 80.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [MemoryConsumption] lower limit : 15.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [LoadAverage] predicted value : 989039.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [LoadAverage] upper limit : 180.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [LoadAverage] lower limit : 20.0
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain scale-up action : true
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain scale-down action : false
> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain [LoadAverage] predicted replicas : 16484
> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> php.php.domain predicted replicas > max replicas :
> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
> scale-up : [cluster] : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] :
> php.php.domain valid number of replicas to expand : 10
> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO
> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
> - Updating kubernetes replication controller via cloud controller:
> [cluster] php.php.domain [replicas] 10
> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Cloud Controller is delegating request to update a replication controller
> php.php.domain to Kubernetes layer.
>
>
> I couldn't investigate the agent side, since containers are destroyed.
>
> Any idea?
>
> Thanks.
>
> --
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
> Mobile +94777568639 | +94783498120
>
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120