You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Rajkumar Rajaratnam <ra...@wso2.com> on 2014/10/13 08:18:39 UTC
CEP removed a terminated member from memberTimeStampMap, but
complaining it as a faulty member
Hi,
CEP extension FaultHandlingWindowProcessor removing a terminated member
from the map. But after a while, it is complaining it as a faulty member.
Here is the log. MemberId highlighted.
TID: [0] [STRATOS] [2014-10-13 11:15:22,409] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
intervals]: 1
TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 41.63388 , [gradient]: -0.013520682 , [second
derivative]: 0.0013238431 , [time intervals]: 1
TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 99.9 , [gradient]: -2.0885546 , [second derivative]:
0.03460533 , [time intervals]: 1
TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [Replicas] minReplicas : 2
TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [Replicas] maxReplicas : 20
TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [Replicas] nonTerminated : 5
TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [Replicas] activeReplicas : 5
TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [RequestInFlight] resetted ? : false
TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [RequestInFlight] predicted value : 0.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [RequestInFlight] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [RequestInFlight] lower limit : 5.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [MemoryConsumption] resetted ? : true
TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [MemoryConsumption] predicted value : 41.62102324969601
TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [MemoryConsumption] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [MemoryConsumption] lower limit : 15.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,415] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [LoadAverage] resetted ? : true
TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [LoadAverage] predicted value : 97.82874980941415
TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [LoadAverage] upper limit : 180.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [LoadAverage] lower limit : 20.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain scale-up action : false
TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain scale-down action : true
TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
scale-down [cluster] : myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-down] [cluster] :
myphp.php.domain shrink the cluster to minReplicas : 2
TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO
{org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
- Updating kubernetes replication controller via cloud controller:
[cluster] myphp.php.domain [replicas] 2
TID: [0] [STRATOS] [2014-10-13 11:15:22,425] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
CloudControllerServiceImpl:updateContainers for cluster : myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:22,448] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Cloud Controller is delegating request to update a replication controller
myphp.php.domain to Kubernetes layer.
TID: [0] [STRATOS] [2014-10-13 11:15:22,506] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Cloud Controller successfully updated the controller myphp.php.domain via
Kubernetes layer.
TID: [0] [STRATOS] [2014-10-13 11:15:22,531] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Pods Count: 5 for cluster: myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:22,548] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:22,561] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:22,563] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:22,655] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:22,657] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:27,538] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
{org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
[b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
Container Activation Watcher Failed.. Cause: Pod
[b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,539] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
{org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
[b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,540] ERROR
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
Container Activation Watcher Failed.. Cause: Pod
[b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,540] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR
{org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
[3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
Container Activation Watcher Failed.. Cause: Pod
[3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,628] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,630] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR
{org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
[b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
Container Activation Watcher Failed.. Cause: Pod
[b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,542] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Pods Count: 2 for cluster: myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Pods created : 2 for cluster : myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
lock acquired
TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
topology
TID: [0] [STRATOS] [2014-10-13 11:15:32,553] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR
{org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
[b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
Container Activation Watcher Failed.. Cause: Pod
[b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,555] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
updated:
{"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
myphp.apachestratos.org
"],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b446c0dd-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b446c0dd-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
lock released
TID: [0] [STRATOS] [2014-10-13 11:15:32,564] INFO
{org.apache.stratos.cloud.controller.topology.TopologyEventPublisher}
- Publishing
member terminated event: [service] php [cluster] myphp.php.domain
[network-partition] null [partition] null [member]
b4468fb6-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,575] INFO
{org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
- [MemberTerminatedEventListener] Received: class
org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
{org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
- Removed the terminated member with id
b4468fb6-529b-11e4-acbb-08002794b041 from the cluster
TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
{org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
- Adding cluster [myphp.php.domain]
TID: [0] [STRATOS] [2014-10-13 11:15:32,580] INFO
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member terminated: [service] php [cluster] myphp.php.domain [member]
b4468fb6-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
{org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
MemberContext [id]: b4468fb6-529b-11e4-acbb-08002794b041 removed from
information model.
TID: [0] [STRATOS] [2014-10-13 11:15:32,581] INFO
{org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
Scheduled Pod Activation Watcher task canceled.
TID: [0] [STRATOS] [2014-10-13 11:15:32,581] WARN
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
is in the wrong list and it is removed from active members list
TID: [0] [STRATOS] [2014-10-13 11:15:32,582] INFO
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
stat context has been removed successfully: [member]
b4468fb6-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,582] WARN
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member already terminated: [service] php [cluster] myphp.php.domain
[member] b4468fb6-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,583] INFO
{org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
[member id] b4468fb6-529b-11e4-acbb-08002794b041 was removed from the time
stamp map.
TID: [0] [STRATOS] [2014-10-13 11:15:32,583] WARN
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member already terminated: [service] php [cluster] myphp.php.domain
[member] b4468fb6-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
lock acquired
TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
topology
TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
updated:
{"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
myphp.apachestratos.org
"],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
lock released
TID: [0] [STRATOS] [2014-10-13 11:15:32,603] INFO
{org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
Publishing member terminated event: [service] php [cluster]
myphp.php.domain [network-partition] null [partition] null [member]
b446c0dd-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,607] INFO
{org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
[member id] b446c0dd-529b-11e4-acbb-08002794b041 was removed from the time
stamp map.
TID: [0] [STRATOS] [2014-10-13 11:15:32,608] INFO
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member terminated: [service] php [cluster] myphp.php.domain [member]
b446c0dd-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO
{org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
- [MemberTerminatedEventListener] Received: class
org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
TID: [0] [STRATOS] [2014-10-13 11:15:32,609] DEBUG
{org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
- Adding cluster [myphp.php.domain]
TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member already terminated: [service] php [cluster] myphp.php.domain
[member] b446c0dd-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
is in the wrong list and it is removed from active members list
TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
stat context has been removed successfully: [member]
b446c0dd-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,610] DEBUG
{org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
MemberContext [id]: b446c0dd-529b-11e4-acbb-08002794b041 removed from
information model.
TID: [0] [STRATOS] [2014-10-13 11:15:32,610] WARN
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member already terminated: [service] php [cluster] myphp.php.domain
[member] b446c0dd-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,610] INFO
{org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
Scheduled Pod Activation Watcher task canceled.
TID: [0] [STRATOS] [2014-10-13 11:15:32,623] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
lock acquired
TID: [0] [STRATOS] [2014-10-13 11:15:32,624] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
topology
TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR
{org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
[3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
Container Activation Watcher Failed.. Cause: Pod
[3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,628] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,633] DEBUG
{org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
updated:
{"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
myphp.apachestratos.org
"],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
lock released
TID: [0] [STRATOS] [2014-10-13 11:15:32,642] INFO
{org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
Publishing member terminated event: [service] php [cluster]
myphp.php.domain [network-partition] null [partition] null [member]
3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,650] INFO
{org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
[member id] 3ccf4e4e-529a-11e4-acbb-08002794b041 was removed from the time
stamp map.
TID: [0] [STRATOS] [2014-10-13 11:15:32,652] INFO
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member terminated: [service] php [cluster] myphp.php.domain [member]
3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,653] INFO
{org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
- [MemberTerminatedEventListener] Received: class
org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
TID: [0] [STRATOS] [2014-10-13 11:15:32,653] DEBUG
{org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
- Adding cluster [myphp.php.domain]
TID: [0] [STRATOS] [2014-10-13 11:15:32,653] WARN
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member already terminated: [service] php [cluster] myphp.php.domain
[member] 3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,654] WARN
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
is in the wrong list and it is removed from active members list
TID: [0] [STRATOS] [2014-10-13 11:15:32,654] DEBUG
{org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
MemberContext [id]: 3ccf4e4e-529a-11e4-acbb-08002794b041 removed from
information model.
TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
stat context has been removed successfully: [member]
3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,655] WARN
{org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
- Member already terminated: [service] php [cluster] myphp.php.domain
[member] 3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO
{org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
Scheduled Pod Activation Watcher task canceled.
TID: [0] [STRATOS] [2014-10-13 11:15:32,696] INFO
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Kubernetes entities are successfully starting up. [MemberContext
[memberId=b4468fb6-529b-11e4-acbb-08002794b041, nodeId=null,
instanceId=null, clusterId=myphp.php.domain, partition=null,
cartridgeType=php, privateIpAddress=172.17.8.100,
publicIpAddress=172.17.8.100, allocatedIpAddress=null,
initTime=1413178952532, lbClusterId=null, networkPartitionId=null,
properties=Properties [properties=[Property
[name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext
[memberId=b446c0dd-529b-11e4-acbb-08002794b041, nodeId=null,
instanceId=null, clusterId=myphp.php.domain, partition=null,
cartridgeType=php, privateIpAddress=172.17.8.101,
publicIpAddress=172.17.8.101, allocatedIpAddress=null,
initTime=1413178952532, lbClusterId=null, networkPartitionId=null,
properties=Properties [properties=[Property
[name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext
[memberId=3ccf4e4e-529a-11e4-acbb-08002794b041, nodeId=null,
instanceId=null, clusterId=myphp.php.domain, partition=null,
cartridgeType=php, privateIpAddress=172.17.8.101,
publicIpAddress=172.17.8.101, allocatedIpAddress=null,
initTime=1413178322623, lbClusterId=null, networkPartitionId=null,
properties=Properties [properties=[Property
[name=ALLOCATED_SERVICE_HOST_PORT, value=4500]]]]]
TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
terminated the member with id b4468fb6-529b-11e4-acbb-08002794b041 in
cluster myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
terminated the member with id b446c0dd-529b-11e4-acbb-08002794b041 in
cluster myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
terminated the member with id 3ccf4e4e-529a-11e4-acbb-08002794b041 in
cluster myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check
executed for :
org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
TID: [0] [STRATOS] [2014-10-13 11:16:01,002] DEBUG
{org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
complete tenant event
TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
Executing topology synchronization task
TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
Publishing complete topology event
TID: [0] [STRATOS] [2014-10-13 11:16:22,230] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] myphp.php.domain [value] -0.70397747
TID: [0] [STRATOS] [2014-10-13 11:16:22,232] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] myphp.php.domain [value] -0.46528494
TID: [0] [STRATOS] [2014-10-13 11:16:22,234] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] myphp.php.domain [value] 27.538944
TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of load avg event: [cluster] myphp.php.domain [value]
-0.0012131054
TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of Memory Consumption event: [cluster] myphp.php.domain [value]
0.021650476
TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] myphp.php.domain [value] 53.7
TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-13 11:16:22,425] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
- Member not found in the Topology: [member]
3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:16:52,406] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
KubernetesServiceClusterMonitor is running..
KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
clusterId=myphp.php.domain, serviceId=php]
TID: [0] [STRATOS] [2014-10-13 11:16:52,407] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
Running min check for [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
[kub-cluster] KubGrp1 [cluster] myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
myphp.php.domain [Replicas] nonTerminated : 2
TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
myphp.php.domain [Replicas] minReplicas : 2
TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
rule [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:16:52,410] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
: myphp.php.domain [Replicas] obsoleteReplicas : 0
TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
check executed for :
org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
flag of rifReset : false flag of memoryConsumptionReset : true flag of
loadAverageReset : true
TID: [0] [STRATOS] [2014-10-13 11:16:52,412] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
Running scale check for [kub-cluster] : KubGrp1 [cluster] :
myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:16:52,412] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
[kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:16:52,414] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
intervals]: 1
TID: [0] [STRATOS] [2014-10-13 11:16:52,415] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 27.538944 , [gradient]: -0.46528494 , [second
derivative]: 0.021650476 , [time intervals]: 1
TID: [0] [STRATOS] [2014-10-13 11:16:52,416] DEBUG
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
value, [average]: 53.7 , [gradient]: -0.70397747 , [second derivative]:
-0.0012131054 , [time intervals]: 1
TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [Replicas] minReplicas : 2
TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [Replicas] maxReplicas : 20
TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [Replicas] nonTerminated : 2
TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [Replicas] activeReplicas : 2
TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [RequestInFlight] resetted ? : false
TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [RequestInFlight] predicted value : 0.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [RequestInFlight] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [RequestInFlight] lower limit : 5.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [MemoryConsumption] resetted ? : true
TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [MemoryConsumption] predicted value : 27.084485135041177
TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [MemoryConsumption] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [MemoryConsumption] lower limit : 15.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,419] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [LoadAverage] resetted ? : true
TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [LoadAverage] predicted value : 52.9954185327515
TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [LoadAverage] upper limit : 180.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain [LoadAverage] lower limit : 20.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain scale-up action : false
TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain scale-down action : true
TID: [0] [STRATOS] [2014-10-13 11:16:52,421] INFO
{org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
myphp.php.domain Even if all predicted stats < upperLimit, still
activeReplicas <= minReplicas ; hence not scaling down
TID: [0] [STRATOS] [2014-10-13 11:16:52,421] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check
executed for :
org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
TID: [0] [STRATOS] [2014-10-13 11:17:01,001] DEBUG
{org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
complete tenant event
TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
Executing topology synchronization task
TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
Publishing complete topology event
TID: [0] [STRATOS] [2014-10-13 11:17:16,141] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last
time-stamp] 1413179137678 [time-out] 60000 milliseconds
TID: [0] [STRATOS] [2014-10-13 11:17:16,141] ERROR
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed
to publish member fault event. Member having [member-id]
3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology
TID: [0] [STRATOS] [2014-10-13 11:17:22,231] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
load avg event: [cluster] myphp.php.domain [value] 23.75
TID: [0] [STRATOS] [2014-10-13 11:17:22,232] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
load avg event: [cluster] myphp.php.domain [value] -0.2682943
TID: [0] [STRATOS] [2014-10-13 11:17:22,236] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
Memory Consumption event: [cluster] myphp.php.domain [value] 17.602432
TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of load avg event: [cluster] myphp.php.domain [value]
-0.0045045386
TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
Memory Consumption event: [cluster] myphp.php.domain [value] 0.0016795773
TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG
{org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
Derivation of Memory Consumption event: [cluster] myphp.php.domain [value]
2.3589536E-4
TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG
{org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-13 11:18:01,000] DEBUG
{org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
complete tenant event
TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
Executing topology synchronization task
TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG
{org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
Publishing complete topology event
TID: [0] [STRATOS] [2014-10-13 11:18:16,142] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last
time-stamp] 1413179137678 [time-out] 60000 milliseconds
TID: [0] [STRATOS] [2014-10-13 11:18:16,144] ERROR
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed
to publish member fault event. Member having [member-id]
3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology
Thanks.
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
Re: CEP removed a terminated member from memberTimeStampMap, but
complaining it as a faulty member
Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
Hi Manula,
We found the issue and fixed it. Problem is updating TimeStampMap without
checking whether the member is in the topology or not.
We are removing member from the map on member terminated event. Also we are
updating the map on receiving health stats message from members. There can
be situation where the health stat message comes bit later than the member
terminated event. In this situation first we are removing member from map
but again we are updating the time stamp map on receiving health stats
messages. It means CEP is still considering it as a active member.
We solved it by checking whether the member is in the topology before
updating time stamp map.
Thanks.
On Mon, Oct 13, 2014 at 2:51 PM, Manula Chathurika Thantriwatte <
manulac@wso2.com> wrote:
> Hi Raj,
>
> It seems like [1] get executed again. memberTimeStampMap is a concurrent
> hash map. Seems that CEPTopologyEventReceiver unable to remove the element
> in the memberTimeStampMap.
>
> [1]
> https://github.com/apache/stratos/blob/master/extensions/cep/stratos-cep-extension/src/main/java/org/apache/stratos/cep/extension/FaultHandlingWindowProcessor.java#L220
>
> Thanks !
>
> On Mon, Oct 13, 2014 at 11:48 AM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Hi,
>>
>> CEP extension FaultHandlingWindowProcessor removing a terminated member
>> from the map. But after a while, it is complaining it as a faulty member.
>> Here is the log. MemberId highlighted.
>>
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,409] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>> intervals]: 1
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 41.63388 , [gradient]: -0.013520682 , [second
>> derivative]: 0.0013238431 , [time intervals]: 1
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 99.9 , [gradient]: -2.0885546 , [second derivative]:
>> 0.03460533 , [time intervals]: 1
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [Replicas] minReplicas : 2
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [Replicas] maxReplicas : 20
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [Replicas] nonTerminated : 5
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [Replicas] activeReplicas : 5
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [RequestInFlight] resetted ? : false
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [RequestInFlight] predicted value : 0.0
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [RequestInFlight] upper limit : 80.0
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [RequestInFlight] lower limit : 5.0
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [MemoryConsumption] resetted ? : true
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [MemoryConsumption] predicted value : 41.62102324969601
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [MemoryConsumption] upper limit : 80.0
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [MemoryConsumption] lower limit : 15.0
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [LoadAverage] resetted ? : true
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [LoadAverage] predicted value : 97.82874980941415
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [LoadAverage] upper limit : 180.0
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [LoadAverage] lower limit : 20.0
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain scale-up action : false
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain scale-down action : true
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
>> scale-down [cluster] : myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-down] [cluster] :
>> myphp.php.domain shrink the cluster to minReplicas : 2
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO
>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
>> - Updating kubernetes replication controller via cloud controller:
>> [cluster] myphp.php.domain [replicas] 2
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,425] DEBUG
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> CloudControllerServiceImpl:updateContainers for cluster : myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,448] DEBUG
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> Cloud Controller is delegating request to update a replication controller
>> myphp.php.domain to Kubernetes layer.
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,506] DEBUG
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> Cloud Controller successfully updated the controller myphp.php.domain via
>> Kubernetes layer.
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,531] DEBUG
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> Pods Count: 5 for cluster: myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,548] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,561] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,563] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,655] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:22,657] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,538] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
>> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
>> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> Container Activation Watcher Failed.. Cause: Pod
>> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
>> Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
>> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
>> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,540] ERROR
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> Container Activation Watcher Failed.. Cause: Pod
>> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,540] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
>> Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR
>> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
>> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> Container Activation Watcher Failed.. Cause: Pod
>> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
>> Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:27,630] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR
>> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
>> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> Container Activation Watcher Failed.. Cause: Pod
>> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
>> Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> Pods Count: 2 for cluster: myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> Pods created : 2 for cluster : myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
>> lock acquired
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
>> topology
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,553] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR
>> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
>> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> Container Activation Watcher Failed.. Cause: Pod
>> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,555] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
>> Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
>> updated:
>> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
>> myphp.apachestratos.org
>> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b446c0dd-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b446c0dd-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
>> lock released
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] INFO
>> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing
>> member terminated event: [service] php [cluster] myphp.php.domain
>> [network-partition] null [partition] null [member]
>> b4468fb6-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,575] INFO
>> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
>> - [MemberTerminatedEventListener] Received: class
>> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
>> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
>> - Removed the terminated member with id
>> b4468fb6-529b-11e4-acbb-08002794b041 from the cluster
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
>> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
>> - Adding cluster [myphp.php.domain]
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] INFO
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member terminated: [service] php [cluster] myphp.php.domain [member]
>> b4468fb6-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
>> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
>> MemberContext [id]: b4468fb6-529b-11e4-acbb-08002794b041 removed from
>> information model.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,581] INFO
>> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
>> Scheduled Pod Activation Watcher task canceled.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,581] WARN
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
>> is in the wrong list and it is removed from active members list
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,582] INFO
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
>> stat context has been removed successfully: [member]
>> b4468fb6-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,582] WARN
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member already terminated: [service] php [cluster] myphp.php.domain
>> [member] b4468fb6-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,583] INFO
>> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
>> [member id] b4468fb6-529b-11e4-acbb-08002794b041 was removed from the time
>> stamp map.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,583] WARN
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member already terminated: [service] php [cluster] myphp.php.domain
>> [member] b4468fb6-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
>> lock acquired
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
>> topology
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
>> updated:
>> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
>> myphp.apachestratos.org
>> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
>> lock released
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] INFO
>> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
>> Publishing member terminated event: [service] php [cluster]
>> myphp.php.domain [network-partition] null [partition] null [member]
>> b446c0dd-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,607] INFO
>> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
>> [member id] b446c0dd-529b-11e4-acbb-08002794b041 was removed from the time
>> stamp map.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,608] INFO
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member terminated: [service] php [cluster] myphp.php.domain [member]
>> b446c0dd-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO
>> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
>> - [MemberTerminatedEventListener] Received: class
>> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] DEBUG
>> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
>> - Adding cluster [myphp.php.domain]
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member already terminated: [service] php [cluster] myphp.php.domain
>> [member] b446c0dd-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
>> is in the wrong list and it is removed from active members list
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
>> stat context has been removed successfully: [member]
>> b446c0dd-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] DEBUG
>> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
>> MemberContext [id]: b446c0dd-529b-11e4-acbb-08002794b041 removed from
>> information model.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] WARN
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member already terminated: [service] php [cluster] myphp.php.domain
>> [member] b446c0dd-529b-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] INFO
>> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
>> Scheduled Pod Activation Watcher task canceled.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,623] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
>> lock acquired
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,624] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
>> topology
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR
>> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
>> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> Container Activation Watcher Failed.. Cause: Pod
>> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,628] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
>> Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,633] DEBUG
>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
>> PodActivationWatcher running : Running
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
>> updated:
>> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
>> myphp.apachestratos.org
>> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
>> lock released
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] INFO
>> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
>> Publishing member terminated event: [service] php [cluster]
>> myphp.php.domain [network-partition] null [partition] null [member]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,650] INFO
>> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
>> [member id] 3ccf4e4e-529a-11e4-acbb-08002794b041 was removed from the time
>> stamp map.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,652] INFO
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member terminated: [service] php [cluster] myphp.php.domain [member]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] INFO
>> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
>> - [MemberTerminatedEventListener] Received: class
>> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] DEBUG
>> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
>> - Adding cluster [myphp.php.domain]
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] WARN
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member already terminated: [service] php [cluster] myphp.php.domain
>> [member] 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] WARN
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
>> is in the wrong list and it is removed from active members list
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] DEBUG
>> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
>> MemberContext [id]: 3ccf4e4e-529a-11e4-acbb-08002794b041 removed from
>> information model.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
>> stat context has been removed successfully: [member]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,655] WARN
>> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
>> - Member already terminated: [service] php [cluster] myphp.php.domain
>> [member] 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO
>> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
>> Scheduled Pod Activation Watcher task canceled.
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,696] INFO
>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
>> Kubernetes entities are successfully starting up. [MemberContext
>> [memberId=b4468fb6-529b-11e4-acbb-08002794b041, nodeId=null,
>> instanceId=null, clusterId=myphp.php.domain, partition=null,
>> cartridgeType=php, privateIpAddress=172.17.8.100,
>> publicIpAddress=172.17.8.100, allocatedIpAddress=null,
>> initTime=1413178952532, lbClusterId=null, networkPartitionId=null,
>> properties=Properties [properties=[Property
>> [name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext
>> [memberId=b446c0dd-529b-11e4-acbb-08002794b041, nodeId=null,
>> instanceId=null, clusterId=myphp.php.domain, partition=null,
>> cartridgeType=php, privateIpAddress=172.17.8.101,
>> publicIpAddress=172.17.8.101, allocatedIpAddress=null,
>> initTime=1413178952532, lbClusterId=null, networkPartitionId=null,
>> properties=Properties [properties=[Property
>> [name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext
>> [memberId=3ccf4e4e-529a-11e4-acbb-08002794b041, nodeId=null,
>> instanceId=null, clusterId=myphp.php.domain, partition=null,
>> cartridgeType=php, privateIpAddress=172.17.8.101,
>> publicIpAddress=172.17.8.101, allocatedIpAddress=null,
>> initTime=1413178322623, lbClusterId=null, networkPartitionId=null,
>> properties=Properties [properties=[Property
>> [name=ALLOCATED_SERVICE_HOST_PORT, value=4500]]]]]
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
>> terminated the member with id b4468fb6-529b-11e4-acbb-08002794b041 in
>> cluster myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
>> terminated the member with id b446c0dd-529b-11e4-acbb-08002794b041 in
>> cluster myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
>> terminated the member with id 3ccf4e4e-529a-11e4-acbb-08002794b041 in
>> cluster myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check
>> executed for :
>> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
>> TID: [0] [STRATOS] [2014-10-13 11:16:01,002] DEBUG
>> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
>> complete tenant event
>> TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
>> Executing topology synchronization task
>> TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
>> Publishing complete topology event
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,230] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] myphp.php.domain [value] -0.70397747
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,232] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> Memory Consumption event: [cluster] myphp.php.domain [value] -0.46528494
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,234] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> Memory Consumption event: [cluster] myphp.php.domain [value] 27.538944
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of load avg event: [cluster] myphp.php.domain [value]
>> -0.0012131054
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of Memory Consumption event: [cluster] myphp.php.domain [value]
>> 0.021650476
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] myphp.php.domain [value] 53.7
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,425] DEBUG
>> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
>> - Member not found in the Topology: [member]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG
>> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
>> - Member not found in the Topology: [member]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG
>> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
>> - Member not found in the Topology: [member]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG
>> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
>> - Member not found in the Topology: [member]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG
>> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
>> - Member not found in the Topology: [member]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,406] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>> KubernetesServiceClusterMonitor is running..
>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
>> clusterId=myphp.php.domain, serviceId=php]
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,407] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>> Running min check for [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
>> [kub-cluster] KubGrp1 [cluster] myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>> myphp.php.domain [Replicas] nonTerminated : 2
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
>> myphp.php.domain [Replicas] minReplicas : 2
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
>> rule [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,410] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
>> : myphp.php.domain [Replicas] obsoleteReplicas : 0
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG
>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
>> check executed for :
>> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>> flag of rifReset : false flag of memoryConsumptionReset : true flag of
>> loadAverageReset : true
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,412] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
>> Running scale check for [kub-cluster] : KubGrp1 [cluster] :
>> myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,412] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
>> [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,414] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
>> intervals]: 1
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,415] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 27.538944 , [gradient]: -0.46528494 , [second
>> derivative]: 0.021650476 , [time intervals]: 1
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,416] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
>> value, [average]: 53.7 , [gradient]: -0.70397747 , [second derivative]:
>> -0.0012131054 , [time intervals]: 1
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [Replicas] minReplicas : 2
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [Replicas] maxReplicas : 20
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [Replicas] nonTerminated : 2
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [Replicas] activeReplicas : 2
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [RequestInFlight] resetted ? : false
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [RequestInFlight] predicted value : 0.0
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [RequestInFlight] upper limit : 80.0
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [RequestInFlight] lower limit : 5.0
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [MemoryConsumption] resetted ? : true
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [MemoryConsumption] predicted value : 27.084485135041177
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [MemoryConsumption] upper limit : 80.0
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [MemoryConsumption] lower limit : 15.0
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] DEBUG
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [LoadAverage] resetted ? : true
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [LoadAverage] predicted value : 52.9954185327515
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [LoadAverage] upper limit : 180.0
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain [LoadAverage] lower limit : 20.0
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain scale-up action : false
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain scale-down action : true
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,421] INFO
>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
>> myphp.php.domain Even if all predicted stats < upperLimit, still
>> activeReplicas <= minReplicas ; hence not scaling down
>> TID: [0] [STRATOS] [2014-10-13 11:16:52,421] DEBUG
>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check
>> executed for :
>> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
>> TID: [0] [STRATOS] [2014-10-13 11:17:01,001] DEBUG
>> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
>> complete tenant event
>> TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
>> Executing topology synchronization task
>> TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
>> Publishing complete topology event
>> TID: [0] [STRATOS] [2014-10-13 11:17:16,141] INFO
>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
>> member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last
>> time-stamp] 1413179137678 [time-out] 60000 milliseconds
>> TID: [0] [STRATOS] [2014-10-13 11:17:16,141] ERROR
>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed
>> to publish member fault event. Member having [member-id]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology
>> TID: [0] [STRATOS] [2014-10-13 11:17:22,231] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> load avg event: [cluster] myphp.php.domain [value] 23.75
>> TID: [0] [STRATOS] [2014-10-13 11:17:22,232] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> load avg event: [cluster] myphp.php.domain [value] -0.2682943
>> TID: [0] [STRATOS] [2014-10-13 11:17:22,236] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
>> Memory Consumption event: [cluster] myphp.php.domain [value] 17.602432
>> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of load avg event: [cluster] myphp.php.domain [value]
>> -0.0045045386
>> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
>> stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
>> Memory Consumption event: [cluster] myphp.php.domain [value] 0.0016795773
>> TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG
>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
>> Derivation of Memory Consumption event: [cluster] myphp.php.domain [value]
>> 2.3589536E-4
>> TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG
>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
>> TID: [0] [STRATOS] [2014-10-13 11:18:01,000] DEBUG
>> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
>> complete tenant event
>> TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
>> Executing topology synchronization task
>> TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG
>> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
>> Publishing complete topology event
>> TID: [0] [STRATOS] [2014-10-13 11:18:16,142] INFO
>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
>> member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last
>> time-stamp] 1413179137678 [time-out] 60000 milliseconds
>> TID: [0] [STRATOS] [2014-10-13 11:18:16,144] ERROR
>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed
>> to publish member fault event. Member having [member-id]
>> 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology
>>
>> Thanks.
>>
>> --
>> Rajkumar Rajaratnam
>> Software Engineer | WSO2, Inc.
>> Mobile +94777568639 | +94783498120
>>
>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manulac@wso2.com / manula@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
>
>
>
--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
Re: CEP removed a terminated member from memberTimeStampMap, but
complaining it as a faulty member
Posted by Manula Chathurika Thantriwatte <ma...@wso2.com>.
Hi Raj,
It seems like [1] get executed again. memberTimeStampMap is a concurrent
hash map. Seems that CEPTopologyEventReceiver unable to remove the element
in the memberTimeStampMap.
[1]
https://github.com/apache/stratos/blob/master/extensions/cep/stratos-cep-extension/src/main/java/org/apache/stratos/cep/extension/FaultHandlingWindowProcessor.java#L220
Thanks !
On Mon, Oct 13, 2014 at 11:48 AM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:
> Hi,
>
> CEP extension FaultHandlingWindowProcessor removing a terminated member
> from the map. But after a while, it is complaining it as a faulty member.
> Here is the log. MemberId highlighted.
>
> TID: [0] [STRATOS] [2014-10-13 11:15:22,409] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
> intervals]: 1
> TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 41.63388 , [gradient]: -0.013520682 , [second
> derivative]: 0.0013238431 , [time intervals]: 1
> TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 99.9 , [gradient]: -2.0885546 , [second derivative]:
> 0.03460533 , [time intervals]: 1
> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [Replicas] minReplicas : 2
> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [Replicas] maxReplicas : 20
> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [Replicas] nonTerminated : 5
> TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [Replicas] activeReplicas : 5
> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [RequestInFlight] resetted ? : false
> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [RequestInFlight] predicted value : 0.0
> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [RequestInFlight] upper limit : 80.0
> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [RequestInFlight] lower limit : 5.0
> TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [MemoryConsumption] resetted ? : true
> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [MemoryConsumption] predicted value : 41.62102324969601
> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [MemoryConsumption] upper limit : 80.0
> TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [MemoryConsumption] lower limit : 15.0
> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [LoadAverage] resetted ? : true
> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [LoadAverage] predicted value : 97.82874980941415
> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [LoadAverage] upper limit : 180.0
> TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [LoadAverage] lower limit : 20.0
> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain scale-up action : false
> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain scale-down action : true
> TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to
> scale-down [cluster] : myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-down] [cluster] :
> myphp.php.domain shrink the cluster to minReplicas : 2
> TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO
> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient}
> - Updating kubernetes replication controller via cloud controller:
> [cluster] myphp.php.domain [replicas] 2
> TID: [0] [STRATOS] [2014-10-13 11:15:22,425] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> CloudControllerServiceImpl:updateContainers for cluster : myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:15:22,448] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Cloud Controller is delegating request to update a replication controller
> myphp.php.domain to Kubernetes layer.
> TID: [0] [STRATOS] [2014-10-13 11:15:22,506] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Cloud Controller successfully updated the controller myphp.php.domain via
> Kubernetes layer.
> TID: [0] [STRATOS] [2014-10-13 11:15:22,531] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Pods Count: 5 for cluster: myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:15:22,548] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:22,561] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:22,563] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:22,655] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:22,657] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:27,538] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> Container Activation Watcher Failed.. Cause: Pod
> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
> Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR
> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,540] ERROR
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> Container Activation Watcher Failed.. Cause: Pod
> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,540] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
> Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR
> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> Container Activation Watcher Failed.. Cause: Pod
> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,628] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
> Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:27,630] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR
> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> Container Activation Watcher Failed.. Cause: Pod
> [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,542] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
> Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Pods Count: 2 for cluster: myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Pods created : 2 for cluster : myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
> lock acquired
> TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
> topology
> TID: [0] [STRATOS] [2014-10-13 11:15:32,553] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR
> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> Container Activation Watcher Failed.. Cause: Pod
> [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,555] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
> Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
> updated:
> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
> myphp.apachestratos.org
> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b446c0dd-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b446c0dd-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
> lock released
> TID: [0] [STRATOS] [2014-10-13 11:15:32,564] INFO
> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - Publishing
> member terminated event: [service] php [cluster] myphp.php.domain
> [network-partition] null [partition] null [member]
> b4468fb6-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,575] INFO
> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
> - [MemberTerminatedEventListener] Received: class
> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
> - Removed the terminated member with id
> b4468fb6-529b-11e4-acbb-08002794b041 from the cluster
> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
> - Adding cluster [myphp.php.domain]
> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] INFO
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member terminated: [service] php [cluster] myphp.php.domain [member]
> b4468fb6-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG
> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
> MemberContext [id]: b4468fb6-529b-11e4-acbb-08002794b041 removed from
> information model.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,581] INFO
> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
> Scheduled Pod Activation Watcher task canceled.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,581] WARN
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
> is in the wrong list and it is removed from active members list
> TID: [0] [STRATOS] [2014-10-13 11:15:32,582] INFO
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
> stat context has been removed successfully: [member]
> b4468fb6-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,582] WARN
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member already terminated: [service] php [cluster] myphp.php.domain
> [member] b4468fb6-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,583] INFO
> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
> [member id] b4468fb6-529b-11e4-acbb-08002794b041 was removed from the time
> stamp map.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,583] WARN
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member already terminated: [service] php [cluster] myphp.php.domain
> [member] b4468fb6-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
> lock acquired
> TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
> topology
> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
> updated:
> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
> myphp.apachestratos.org
> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
> lock released
> TID: [0] [STRATOS] [2014-10-13 11:15:32,603] INFO
> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
> Publishing member terminated event: [service] php [cluster]
> myphp.php.domain [network-partition] null [partition] null [member]
> b446c0dd-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,607] INFO
> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
> [member id] b446c0dd-529b-11e4-acbb-08002794b041 was removed from the time
> stamp map.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,608] INFO
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member terminated: [service] php [cluster] myphp.php.domain [member]
> b446c0dd-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO
> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
> - [MemberTerminatedEventListener] Received: class
> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] DEBUG
> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
> - Adding cluster [myphp.php.domain]
> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member already terminated: [service] php [cluster] myphp.php.domain
> [member] b446c0dd-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
> is in the wrong list and it is removed from active members list
> TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
> stat context has been removed successfully: [member]
> b446c0dd-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] DEBUG
> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
> MemberContext [id]: b446c0dd-529b-11e4-acbb-08002794b041 removed from
> information model.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] WARN
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member already terminated: [service] php [cluster] myphp.php.domain
> [member] b446c0dd-529b-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,610] INFO
> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
> Scheduled Pod Activation Watcher task canceled.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,623] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
> lock acquired
> TID: [0] [STRATOS] [2014-10-13 11:15:32,624] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updating
> topology
> TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR
> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod
> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> Container Activation Watcher Failed.. Cause: Pod
> [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,628] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
> Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,633] DEBUG
> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -
> PodActivationWatcher running : Running
> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topology
> updated:
> {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["
> myphp.apachestratos.org
> "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false}
> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write
> lock released
> TID: [0] [STRATOS] [2014-10-13 11:15:32,642] INFO
> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
> Publishing member terminated event: [service] php [cluster]
> myphp.php.domain [network-partition] null [partition] null [member]
> 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,650] INFO
> {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member
> [member id] 3ccf4e4e-529a-11e4-acbb-08002794b041 was removed from the time
> stamp map.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,652] INFO
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member terminated: [service] php [cluster] myphp.php.domain [member]
> 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] INFO
> {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver}
> - [MemberTerminatedEventListener] Received: class
> org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] DEBUG
> {org.apache.stratos.manager.topology.model.TopologyClusterInformationModel}
> - Adding cluster [myphp.php.domain]
> TID: [0] [STRATOS] [2014-10-13 11:15:32,653] WARN
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member already terminated: [service] php [cluster] myphp.php.domain
> [member] 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] WARN
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
> is in the wrong list and it is removed from active members list
> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] DEBUG
> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
> MemberContext [id]: 3ccf4e4e-529a-11e4-acbb-08002794b041 removed from
> information model.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Member
> stat context has been removed successfully: [member]
> 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,655] WARN
> {org.apache.stratos.messaging.message.processor.topology.MemberTerminatedMessageProcessor}
> - Member already terminated: [service] php [cluster] myphp.php.domain
> [member] 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO
> {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} -
> Scheduled Pod Activation Watcher task canceled.
> TID: [0] [STRATOS] [2014-10-13 11:15:32,696] INFO
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Kubernetes entities are successfully starting up. [MemberContext
> [memberId=b4468fb6-529b-11e4-acbb-08002794b041, nodeId=null,
> instanceId=null, clusterId=myphp.php.domain, partition=null,
> cartridgeType=php, privateIpAddress=172.17.8.100,
> publicIpAddress=172.17.8.100, allocatedIpAddress=null,
> initTime=1413178952532, lbClusterId=null, networkPartitionId=null,
> properties=Properties [properties=[Property
> [name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext
> [memberId=b446c0dd-529b-11e4-acbb-08002794b041, nodeId=null,
> instanceId=null, clusterId=myphp.php.domain, partition=null,
> cartridgeType=php, privateIpAddress=172.17.8.101,
> publicIpAddress=172.17.8.101, allocatedIpAddress=null,
> initTime=1413178952532, lbClusterId=null, networkPartitionId=null,
> properties=Properties [properties=[Property
> [name=ALLOCATED_SERVICE_HOST_PORT, value=null]]]], MemberContext
> [memberId=3ccf4e4e-529a-11e4-acbb-08002794b041, nodeId=null,
> instanceId=null, clusterId=myphp.php.domain, partition=null,
> cartridgeType=php, privateIpAddress=172.17.8.101,
> publicIpAddress=172.17.8.101, allocatedIpAddress=null,
> initTime=1413178322623, lbClusterId=null, networkPartitionId=null,
> properties=Properties [properties=[Property
> [name=ALLOCATED_SERVICE_HOST_PORT, value=4500]]]]]
> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
> terminated the member with id b4468fb6-529b-11e4-acbb-08002794b041 in
> cluster myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
> terminated the member with id b446c0dd-529b-11e4-acbb-08002794b041 in
> cluster myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down,
> terminated the member with id 3ccf4e4e-529a-11e4-acbb-08002794b041 in
> cluster myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG
> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check
> executed for :
> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
> TID: [0] [STRATOS] [2014-10-13 11:16:01,002] DEBUG
> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
> complete tenant event
> TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
> Executing topology synchronization task
> TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
> Publishing complete topology event
> TID: [0] [STRATOS] [2014-10-13 11:16:22,230] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] myphp.php.domain [value] -0.70397747
> TID: [0] [STRATOS] [2014-10-13 11:16:22,232] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] myphp.php.domain [value] -0.46528494
> TID: [0] [STRATOS] [2014-10-13 11:16:22,234] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] myphp.php.domain [value] 27.538944
> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of load avg event: [cluster] myphp.php.domain [value]
> -0.0012131054
> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of Memory Consumption event: [cluster] myphp.php.domain [value]
> 0.021650476
> TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] myphp.php.domain [value] 53.7
> TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-13 11:16:22,425] DEBUG
> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
> - Member not found in the Topology: [member]
> 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG
> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
> - Member not found in the Topology: [member]
> 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG
> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
> - Member not found in the Topology: [member]
> 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG
> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
> - Member not found in the Topology: [member]
> 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG
> {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
> - Member not found in the Topology: [member]
> 3ccf4e4e-529a-11e4-acbb-08002794b041
> TID: [0] [STRATOS] [2014-10-13 11:16:52,406] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> KubernetesServiceClusterMonitor is running..
> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1,
> clusterId=myphp.php.domain, serviceId=php]
> TID: [0] [STRATOS] [2014-10-13 11:16:52,407] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> Running min check for [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule:
> [kub-cluster] KubGrp1 [cluster] myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
> myphp.php.domain [Replicas] nonTerminated : 2
> TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] :
> myphp.php.domain [Replicas] minReplicas : 2
> TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers
> rule [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:16:52,410] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster]
> : myphp.php.domain [Replicas] obsoleteReplicas : 0
> TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG
> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum
> check executed for :
> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
> TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> flag of rifReset : false flag of memoryConsumptionReset : true flag of
> loadAverageReset : true
> TID: [0] [STRATOS] [2014-10-13 11:16:52,412] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -
> Running scale check for [kub-cluster] : KubGrp1 [cluster] :
> myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:16:52,412] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule
> [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain
> TID: [0] [STRATOS] [2014-10-13 11:16:52,414] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time
> intervals]: 1
> TID: [0] [STRATOS] [2014-10-13 11:16:52,415] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 27.538944 , [gradient]: -0.46528494 , [second
> derivative]: 0.021650476 , [time intervals]: 1
> TID: [0] [STRATOS] [2014-10-13 11:16:52,416] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the
> value, [average]: 53.7 , [gradient]: -0.70397747 , [second derivative]:
> -0.0012131054 , [time intervals]: 1
> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [Replicas] minReplicas : 2
> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [Replicas] maxReplicas : 20
> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [Replicas] nonTerminated : 2
> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [Replicas] activeReplicas : 2
> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [RequestInFlight] resetted ? : false
> TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [RequestInFlight] predicted value : 0.0
> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [RequestInFlight] upper limit : 80.0
> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [RequestInFlight] lower limit : 5.0
> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [MemoryConsumption] resetted ? : true
> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [MemoryConsumption] predicted value : 27.084485135041177
> TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [MemoryConsumption] upper limit : 80.0
> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [MemoryConsumption] lower limit : 15.0
> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] DEBUG
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [LoadAverage] resetted ? : true
> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [LoadAverage] predicted value : 52.9954185327515
> TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [LoadAverage] upper limit : 180.0
> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain [LoadAverage] lower limit : 20.0
> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain scale-up action : false
> TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain scale-down action : true
> TID: [0] [STRATOS] [2014-10-13 11:16:52,421] INFO
> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] :
> myphp.php.domain Even if all predicted stats < upperLimit, still
> activeReplicas <= minReplicas ; hence not scaling down
> TID: [0] [STRATOS] [2014-10-13 11:16:52,421] DEBUG
> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale check
> executed for :
> org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84
> TID: [0] [STRATOS] [2014-10-13 11:17:01,001] DEBUG
> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
> complete tenant event
> TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
> Executing topology synchronization task
> TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
> Publishing complete topology event
> TID: [0] [STRATOS] [2014-10-13 11:17:16,141] INFO
> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
> member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last
> time-stamp] 1413179137678 [time-out] 60000 milliseconds
> TID: [0] [STRATOS] [2014-10-13 11:17:16,141] ERROR
> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed
> to publish member fault event. Member having [member-id]
> 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology
> TID: [0] [STRATOS] [2014-10-13 11:17:22,231] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> load avg event: [cluster] myphp.php.domain [value] 23.75
> TID: [0] [STRATOS] [2014-10-13 11:17:22,232] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> load avg event: [cluster] myphp.php.domain [value] -0.2682943
> TID: [0] [STRATOS] [2014-10-13 11:17:22,236] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg
> Memory Consumption event: [cluster] myphp.php.domain [value] 17.602432
> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of load avg event: [cluster] myphp.php.domain [value]
> -0.0045045386
> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average
> stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of
> Memory Consumption event: [cluster] myphp.php.domain [value] 0.0016795773
> TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG
> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second
> Derivation of Memory Consumption event: [cluster] myphp.php.domain [value]
> 2.3589536E-4
> TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG
> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory
> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
> TID: [0] [STRATOS] [2014-10-13 11:18:01,000] DEBUG
> {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishing
> complete tenant event
> TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} -
> Executing topology synchronization task
> TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG
> {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} -
> Publishing complete topology event
> TID: [0] [STRATOS] [2014-10-13 11:18:16,142] INFO
> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
> member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last
> time-stamp] 1413179137678 [time-out] 60000 milliseconds
> TID: [0] [STRATOS] [2014-10-13 11:18:16,144] ERROR
> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed
> to publish member fault event. Member having [member-id]
> 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology
>
> Thanks.
>
> --
> Rajkumar Rajaratnam
> Software Engineer | WSO2, Inc.
> Mobile +94777568639 | +94783498120
>
--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware
email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/