You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@stratos.apache.org by Imesh Gunaratne <im...@apache.org> on 2015/01/01 07:31:35 UTC

Re: [Discuss] Kubernetes workflow is now working!

A good point Raj, yes, it is not consistent to let the cloud controller
handle this logic as it is already done by the Autoscaler. I will fix this.

Thanks

On Wed, Dec 31, 2014 at 8:51 PM, Rajkumar Rajaratnam <ra...@wso2.com>
wrote:

> Hi,
>
> Currently pods' status are monitored by CC. If pods are not getting
> "Running" status within a time period(default to 120s), replication
> controllers and pods are deleted by CC and nothing happens after that.
>
> I faced some problems due to this. My pods and replication controller were
> deleted by CC, because pods didn't come to "Running" status.
>
> ERROR
> {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} -
> Pod status did not change to running within 120 sec, hence removing
> replication controller and pod: [cluster-id] php.php.domain [member-id]
> php.php.domaine9f27b1c-6694-4b70-b939-f80b2ad74168
> [replication-controller-id]
> php-php-domaine9f27b1c-6694-4b70-b939-f80b2ad74168 [pod-id]
> php-php-domaine9f27b1c-6694-4b70-b939-f80b2ad74168
>
> But AS doesn't know about this. Later AS moved these members to obsolete
> list. When AS asks CC to terminate these pods, CC throws exception saying
> member context not found.
>
>
>
>
>
> *rajkumar@rajkumar-Latitude-E6540:/opt/stratos-packs/apache-stratos-4.1.0-SNAPSHOT/bin$
> [2014-12-31 20:02:16,728]  INFO
> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}
> -  Pending state of member:
> php.php.domaine9f27b1c-6694-4b70-b939-f80b2ad74168 is expired. Adding as an
> obsoleted member.[2014-12-31 20:03:31,788]  INFO
> {org.apache.stratos.autoscaler.client.CloudControllerClient} -  Terminating
> instance via cloud controller: [member]
> php.php.domaine9f27b1c-6694-4b70-b939-f80b2ad74168[2014-12-31 20:03:31,798]
> ERROR
> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
> -  Member termination failed, member context not found: [member-id]
> php.php.domaine9f27b1c-6694-4b70-b939-f80b2ad74168[2014-12-31 20:03:31,799]
> ERROR {org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} -  Member
> termination failed, member context not found: [member-id]
> php.php.domaine9f27b1c-6694-4b70-b939-f80b2ad74168org.apache.stratos.cloud.controller.exception.InvalidMemberException:
> Member termination failed, member context not found: [member-id]
> php.php.domaine9f27b1c-6694-4b70-b939-f80b2ad74168*
>
> Is this a correct flow?
>
> IMO, we don't need to monitor the pods' status at CC side. Rather, AS will
> move these pods to obsolete list once the pending time is expired. Then AS
> will ask CC to delete these pods. This is how VM scenario works now.
> Otherwise, CC and AS will not be in same state. For example, AS will ask to
> terminate a member which CC does not have.
>
> One more thing I noticed that member context is added to CC's data holder
> only after pods are coming to "Running" status. IMO, we need to add the
> member context as soon as pods are created. This ensures that when AS asks
> a member to terminate, CC will always have it.
>
> wdyt?
>
> Thanks.
>
> On Wed, Dec 31, 2014 at 7:47 PM, Rajkumar Rajaratnam <ra...@wso2.com>
> wrote:
>
>> Thanks Dakshika,
>>
>> Yes my stratos-installer was outdated. Its working fine now.
>>
>> Thanks.
>>
>> On Tue, Dec 30, 2014 at 11:35 AM, Dakshika Jayathilaka <dakshika@wso2.com
>> > wrote:
>>
>>> Hi Raj,
>>>
>>> seems you  missed below DB sqls,  reside on *tools/stratos-*
>>> *installer/resources*.
>>>
>>> application-mysql.sql
>>> security-mysql.sql
>>>
>>> execute both and retry..
>>>
>>> Thank you,
>>>
>>> Regards,
>>>
>>> *Dakshika Jayathilaka*
>>> Software Engineer
>>> WSO2, Inc.
>>> lean.enterprise.middleware
>>> 0771100911
>>>
>>> On Tue, Dec 30, 2014 at 9:07 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying our kubernetes. I am getting the following exception while
>>>> deploying an application.
>>>>
>>>> [2014-12-30 09:02:14,414]  INFO
>>>> {org.apache.stratos.autoscaler.api.AutoScalerServiceImpl} -  Adding
>>>> application: [application-id] single-cartridge-app
>>>> [2014-12-30 09:02:14,482]  INFO
>>>> {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} -
>>>> 'admin@carbon.super [-1234]' logged in at [2014-12-30
>>>> 09:02:14,482+0530]
>>>> [2014-12-30 09:02:14,492] ERROR
>>>> {org.wso2.carbon.identity.oauth.dao.OAuthAppDAO} -  Error when executing
>>>> the SQL : SELECT * FROM IDN_OAUTH_CONSUMER_APPS WHERE USERNAME=? AND
>>>> TENANT_ID=? AND APP_NAME=?
>>>> [2014-12-30 09:02:14,493] ERROR
>>>> {org.wso2.carbon.identity.oauth.dao.OAuthAppDAO} -  Table
>>>> 'userstore.IDN_OAUTH_CONSUMER_APPS' doesn't exist
>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
>>>> 'userstore.IDN_OAUTH_CONSUMER_APPS' doesn't exist
>>>>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>> Method)
>>>>     at
>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>     at
>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>     at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>     at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
>>>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
>>>>     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
>>>>     at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
>>>>     at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
>>>>     at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2834)
>>>>     at
>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
>>>>     at
>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
>>>>     at
>>>> org.wso2.carbon.identity.oauth.dao.OAuthAppDAO.isDuplicateApplication(OAuthAppDAO.java:372)
>>>>     at
>>>> org.wso2.carbon.identity.oauth.dao.OAuthAppDAO.addOAuthApplication(OAuthAppDAO.java:67)
>>>>     at
>>>> org.wso2.carbon.identity.oauth.OAuthAdminService.registerOAuthApplicationData(OAuthAdminService.java:228)
>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>     at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>     at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>>     at
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>>     at
>>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>>>     at
>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>>     at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>>     at
>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>>     at
>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>>     at
>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>     at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>>     at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>>     at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>>     at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>     at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>>     at
>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>>     at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>>     at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>     at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>>     at
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>>     at
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>>     at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>     at java.lang.Thread.run(Thread.java:662)
>>>> [2014-12-30 09:02:14,496] ERROR
>>>> {org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} -  Error when
>>>> reading the application information from the persistence store.
>>>> org.wso2.carbon.identity.oauth.IdentityOAuthAdminException: Error when
>>>> reading the application information from the persistence store.
>>>>     at
>>>> org.wso2.carbon.identity.oauth.dao.OAuthAppDAO.isDuplicateApplication(OAuthAppDAO.java:383)
>>>>     at
>>>> org.wso2.carbon.identity.oauth.dao.OAuthAppDAO.addOAuthApplication(OAuthAppDAO.java:67)
>>>>     at
>>>> org.wso2.carbon.identity.oauth.OAuthAdminService.registerOAuthApplicationData(OAuthAdminService.java:228)
>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>     at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>     at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>>     at
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>>     at
>>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>>>     at
>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>>     at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>>     at
>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>>     at
>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>>     at
>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>     at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>>     at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>>     at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>>     at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>     at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>>     at
>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>>     at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>>     at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>     at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>>     at
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>>     at
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>>     at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>     at java.lang.Thread.run(Thread.java:662)
>>>> [2014-12-30 09:02:14,511] ERROR
>>>> {org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} -
>>>> org.apache.axis2.AxisFault: Error when reading the application information
>>>> from the persistence store.
>>>> org.apache.stratos.autoscaler.exception.AutoScalerException:
>>>> org.apache.axis2.AxisFault: Error when reading the application information
>>>> from the persistence store.
>>>>     at
>>>> org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser.createToken(DefaultApplicationParser.java:714)
>>>>     at
>>>> org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser.parse(DefaultApplicationParser.java:109)
>>>>     at
>>>> org.apache.stratos.autoscaler.api.AutoScalerServiceImpl.addApplication(AutoScalerServiceImpl.java:173)
>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>     at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>     at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>>     at
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>>     at
>>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>>>     at
>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>>     at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>>     at
>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>>     at
>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>>     at
>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>>>     at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>     at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>>     at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>>     at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>>     at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>     at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>>     at
>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>>     at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>>     at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>     at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>>     at
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>>     at
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>>     at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>     at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: org.apache.axis2.AxisFault: Error when reading the
>>>> application information from the persistence store.
>>>>     at
>>>> org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
>>>>     at
>>>> org.apache.axis2.description.RobustOutOnlyAxisOperation$RobustOutOnlyOperationClient.handleResponse(RobustOutOnlyAxisOperation.java:91)
>>>>     at
>>>> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:445)
>>>>     at
>>>> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
>>>>     at
>>>> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
>>>>     at
>>>> org.wso2.carbon.identity.oauth.stub.OAuthAdminServiceStub.registerOAuthApplicationData(OAuthAdminServiceStub.java:323)
>>>>     at
>>>> org.apache.stratos.autoscaler.client.oAuthAdminServiceClient.registerOauthApplication(oAuthAdminServiceClient.java:97)
>>>>     at
>>>> org.apache.stratos.autoscaler.applications.parser.DefaultApplicationParser.createToken(DefaultApplicationParser.java:712)
>>>>     ... 46 more
>>>> [2014-12-30 09:02:14,521] ERROR
>>>> {org.apache.stratos.rest.endpoint.handlers.CustomExceptionMapper} -  Error
>>>> while fulfilling the request
>>>> org.apache.stratos.rest.endpoint.exception.RestAPIException
>>>>     at
>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addApplication(StratosApiV41Utils.java:974)
>>>>     at
>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41.addApplication(StratosApiV41.java:354)
>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>     at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>     at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>>     at
>>>> org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
>>>>     at
>>>> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
>>>>     at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:194)
>>>>     at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
>>>>     at
>>>> org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
>>>>     at
>>>> org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
>>>>     at
>>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
>>>>     at
>>>> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
>>>>     at
>>>> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
>>>>     at
>>>> org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223)
>>>>     at
>>>> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203)
>>>>     at
>>>> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)
>>>>     at
>>>> org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:159)
>>>>     at
>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
>>>>     at
>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>>     at
>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>>     at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>     at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>>     at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>>     at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>>     at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>     at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>>     at
>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>>     at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>>     at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>>     at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>     at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>>     at
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>>     at
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>>     at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>     at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>     at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: org.apache.axis2.AxisFault: org.apache.axis2.AxisFault:
>>>> Error when reading the application information from the persistence store.
>>>>     at
>>>> org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
>>>>     at
>>>> org.apache.axis2.description.RobustOutOnlyAxisOperation$RobustOutOnlyOperationClient.handleResponse(RobustOutOnlyAxisOperation.java:91)
>>>>     at
>>>> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:445)
>>>>     at
>>>> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
>>>>     at
>>>> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
>>>>     at
>>>> org.apache.stratos.autoscaler.stub.AutoScalerServiceStub.addApplication(AutoScalerServiceStub.java:3223)
>>>>     at
>>>> org.apache.stratos.manager.client.AutoscalerServiceClient.addApplication(AutoscalerServiceClient.java:122)
>>>>     at
>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addApplication(StratosApiV41Utils.java:970)
>>>>     ...
>>>>
>>>> Any idea?
>>>>
>>>> Thanks.
>>>>
>>>> On Mon, Dec 29, 2014 at 4:06 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> I did an improvement in the container startup logic to wait until the
>>>>> pod status changes to Runnning state. Here cloud controller will wait until
>>>>> two minutes to a pod to become active. if not the call will be rolled back.
>>>>> Pod activation timeout can be configured using the system property:
>>>>> stratos.pod.activation.timeout.
>>>>>
>>>>> Once pod changes to Running state, IP addresses will be updated as
>>>>> follows:
>>>>> - Member's default private IP: Pod's IP address
>>>>> - Member's default public IP: Pod's host IP address
>>>>>
>>>>> A sample log:
>>>>>
>>>>> [2014-12-29 15:43:02,419]  INFO Starting container: [cartridge-type]
>>>>> php
>>>>> [2014-12-29 15:43:02,454]  INFO Creating replication controller:
>>>>> [replication-controller-id]
>>>>> php-php-domainfc7e626a-e1e0-456d-8528-744f256c1f7a [name]
>>>>> php-php-domainfc7e626a-e1e0-456d-8528-744f256c1f7a [docker-image]
>>>>> stratos/php:4.1.0-alpha [container-ports] [80] [replicas] 1
>>>>> [2014-12-29 15:43:02,559]  INFO Replication controller created
>>>>> successfully: [replication-controller-id]
>>>>> php-php-domainfc7e626a-e1e0-456d-8528-744f256c1f7a [name]
>>>>> php-php-domainfc7e626a-e1e0-456d-8528-744f256c1f7a [docker-image]
>>>>> stratos/php:4.1.0-alpha [container-port] [80] [replicas] 1
>>>>> [2014-12-29 15:43:02,560]  INFO Creating service: [service-id]
>>>>> php-php-domain-http-80 [service-name] php-php-domain-http-80 [service-port]
>>>>> 4500 [container-port] 80, [public-ip] 172.17.8.100
>>>>> [2014-12-29 15:43:02,598]  INFO Service created successfully:
>>>>> [service-id] php-php-domain-http-80 [service-name] php-php-domain-http-80
>>>>> [service-port] 4500 [container-port] 80, [public-ip] 172.17.8.100
>>>>> [2014-12-29 15:43:07,412]  *INFO Waiting pod status to be changed to
>>>>> running*: [member-id]
>>>>> php.php.domainfc7e626a-e1e0-456d-8528-744f256c1f7a [pod-id]
>>>>> 440b7235-8f43-11e4-b193-080027f35f1b [current-pod-status] waiting
>>>>> [2014-12-29 15:43:18,459]  *INFO Waiting pod status to be changed to
>>>>> running*: [member-id]
>>>>> php.php.domainfc7e626a-e1e0-456d-8528-744f256c1f7a [pod-id]
>>>>> 440b7235-8f43-11e4-b193-080027f35f1b [current-pod-status] waiting
>>>>> [2014-12-29 15:43:29,509]  *INFO Pod status changed to running*:
>>>>> [member-id] php.php.domainfc7e626a-e1e0-456d-8528-744f256c1f7a [pod-id]
>>>>> 440b7235-8f43-11e4-b193-080027f35f1b
>>>>> [2014-12-29 15:43:33,595]  INFO Instance started successfully:
>>>>> [cartridge-type] php [cluster-id] php.php.domain [instance-id]
>>>>> 440b7235-8f43-11e4-b193-080027f35f1b [default-private-ip] 10.100.37.4
>>>>> [default-public-ip] 172.17.8.100
>>>>> [2014-12-29 15:43:33,601]  INFO Publishing instance spawned event:
>>>>> [service-name] php [cluster-id] php.php.domain [cluster-instance-id]
>>>>> single-cartridge-app-1 [member-id]
>>>>> php.php.domainfc7e626a-e1e0-456d-8528-744f256c1f7a [instance-id]
>>>>> 440b7235-8f43-11e4-b193-080027f35f1b [network-partition-id] network-p1
>>>>> [partition-id] p1 [lb-cluster-id] null
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Sun, Dec 28, 2014 at 7:40 PM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thanks Lakmal!
>>>>>> After the application becomes active I can see that Autoscaler is
>>>>>> detecting a fauly member:
>>>>>>
>>>>>> [2014-12-28 19:12:14,016]  INFO
>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} -  Faulty
>>>>>> member detected [member-id]
>>>>>> php.php.domaincc7fb2e4-ea96-4b70-8706-353e4225f170 with [last time-stamp]
>>>>>> 1419774066943 [time-out] 60000 milliseconds
>>>>>> [2014-12-28 19:12:14,025]  INFO
>>>>>> {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} -
>>>>>>  Publishing member fault event for [member-id]
>>>>>> php.php.domaincc7fb2e4-ea96-4b70-8706-353e4225f170
>>>>>> [2014-12-28 19:12:14,036]  INFO
>>>>>> {org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusInactiveProcessor}
>>>>>> -  Publishing Cluster inactivate event for [application]:
>>>>>> single-cartridge-app [cluster]: php.php.domain
>>>>>>
>>>>>> I'm currently investigating the reason for this.
>>>>>>
>>>>>> On Fri, Dec 26, 2014 at 7:09 PM, Lakmal Warusawithana <
>>>>>> lakmal@wso2.com> wrote:
>>>>>>
>>>>>>> thanks Imesh, will give a try.
>>>>>>>
>>>>>>> On Fri, Dec 26, 2014 at 6:35 PM, Imesh Gunaratne <im...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> After fixing the above issue, the application is getting active!
>>>>>>>> :-) I used a single cartridge application for this test. Now we can create
>>>>>>>> a more complex application and see how it works.
>>>>>>>>
>>>>>>>> I have written a small guide on describing all steps for trying out
>>>>>>>> Kubernetes workflow, please try out:
>>>>>>>> https://gist.github.com/imesh/b8f81fac8de39183a504
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Fri, Dec 26, 2014 at 5:25 PM, Imesh Gunaratne <im...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I think it should be as follows:
>>>>>>>>>
>>>>>>>>> Member private IP => container IP
>>>>>>>>> Member default private IP => container IP
>>>>>>>>> Member public IPs = > service proxy IPs
>>>>>>>>> Member default public IP => kubernetes master/host IP
>>>>>>>>>
>>>>>>>>> If public IP assignment is not enabled service proxies are not
>>>>>>>>> created, I can recall that Lakmal suggested this.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Fri, Dec 26, 2014 at 5:10 PM, Imesh Gunaratne <imesh@apache.org
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> I fixed the above issue by setting the private and public IPs to
>>>>>>>>>> Kubernetes Master IP, we will need to review few things here:
>>>>>>>>>> - Private IP address of a member needs to be the service proxy IP
>>>>>>>>>> address
>>>>>>>>>> - However each service proxy will have a unique IP address (each
>>>>>>>>>> transport of the server)
>>>>>>>>>> - Which means there will be a private IP for each transport
>>>>>>>>>>
>>>>>>>>>> Will discuss this and see how we can resolve this problem later
>>>>>>>>>> on. For the moment setting the Kubernetes Master IP is sufficient IMO.
>>>>>>>>>>
>>>>>>>>>> After fixing the above issue, now I see the below error:
>>>>>>>>>>
>>>>>>>>>> [2014-12-26 17:05:30,547] ERROR
>>>>>>>>>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
>>>>>>>>>> -  Error processing event null
>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>> at
>>>>>>>>>> org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor.getClusterInstanceContext(ClusterMonitor.java:1116)
>>>>>>>>>> at
>>>>>>>>>> org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor.handleMemberActivatedEvent(ClusterMonitor.java:875)
>>>>>>>>>> at
>>>>>>>>>> org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver$12.onEvent(AutoscalerTopologyEventReceiver.java:393)
>>>>>>>>>> at
>>>>>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 26, 2014 at 4:56 PM, Rajkumar Rajaratnam <
>>>>>>>>>> rajkumarr@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 26, 2014 at 4:34 PM, Imesh Gunaratne <
>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Lakmal,
>>>>>>>>>>>>
>>>>>>>>>>>> I added these to Kubernetes Cluster definition for the time
>>>>>>>>>>>> being since I'm finding/fixing many issues in agent and docker base image,
>>>>>>>>>>>> will discuss and move them to a proper location later on. WDYT?
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>       "clusterId": "kubernetes-cluster-1",
>>>>>>>>>>>>       "description": "Kubernetes CoreOS cluster",
>>>>>>>>>>>>       ...
>>>>>>>>>>>>     "property":[
>>>>>>>>>>>>        {
>>>>>>>>>>>>           "name":"payload_parameter.MB_IP",
>>>>>>>>>>>>           "value":"192.168.59.3"
>>>>>>>>>>>>        },
>>>>>>>>>>>>        {
>>>>>>>>>>>>           "name":"payload_parameter.MB_PORT",
>>>>>>>>>>>>           "value":"1883"
>>>>>>>>>>>>        },
>>>>>>>>>>>>        {
>>>>>>>>>>>>           "name":"payload_parameter.CEP_IP",
>>>>>>>>>>>>           "value":"192.168.59.3"
>>>>>>>>>>>>        },
>>>>>>>>>>>>        {
>>>>>>>>>>>>           "name":"payload_parameter.CEP_PORT",
>>>>>>>>>>>>           "value":"7711"
>>>>>>>>>>>>        },
>>>>>>>>>>>>        {
>>>>>>>>>>>>           "name":"payload_parameter.LOG_LEVEL",
>>>>>>>>>>>>           "value":"DEBUG"
>>>>>>>>>>>>        }
>>>>>>>>>>>>     ]
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> After the above fix I'm now seeing the below error in cloud
>>>>>>>>>>>> controller. Seems like we do not send the ip address in the member
>>>>>>>>>>>> activated event. Need to check agent logic again:
>>>>>>>>>>>>
>>>>>>>>>>>> [2014-12-26 16:28:06,218]  INFO
>>>>>>>>>>>> {org.apache.stratos.manager.listener.InstanceStatusListener} -  Instance
>>>>>>>>>>>> status message received
>>>>>>>>>>>> [2014-12-26 16:28:06,218]  INFO
>>>>>>>>>>>> {org.apache.stratos.manager.listener.InstanceStatusListener} -  Event class
>>>>>>>>>>>> name:
>>>>>>>>>>>> org.apache.stratos.messaging.event.instance.status.InstanceActivatedEvent
>>>>>>>>>>>> [2014-12-26 16:28:06,219]  INFO
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>>>>>>>>>>>  member started event adding status activated
>>>>>>>>>>>> [2014-12-26 16:28:06,225]  INFO
>>>>>>>>>>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>>>>>>>>>>> -  Publishing member activated event: [service-name] php [cluster-id]
>>>>>>>>>>>> php.php.domain [cluster-instance-id] single-cartridge-app-1 [member-id]
>>>>>>>>>>>> php.php.domaind7655e6f-f962-425b-b93b-2bf7333dc37d [network-partition-id]
>>>>>>>>>>>>  [partition-id]
>>>>>>>>>>>> [2014-12-26 16:28:06,244] ERROR
>>>>>>>>>>>> {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
>>>>>>>>>>>> -  Failed to retrieve topology event message
>>>>>>>>>>>> java.lang.RuntimeException: *No ip address found in member
>>>>>>>>>>>> activated event: *[service] php [cluster] php.php.domain
>>>>>>>>>>>> [member] php.php.domaind7655e6f-f962-425b-b93b-2bf7333dc37d
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.stratos.messaging.message.processor.topology.MemberActivatedMessageProcessor.doProcess(MemberActivatedMessageProcessor.java:102)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I guess spawned containers doesn't have IP addresses. How we are
>>>>>>>>>>> getting IP address for a container is like this.
>>>>>>>>>>>
>>>>>>>>>>> // PodActivationWatcher.java
>>>>>>>>>>> String hostIP = pod.getCurrentState().getHost();
>>>>>>>>>>>
>>>>>>>>>>> This is returning null or empty, so member activated event is
>>>>>>>>>>> not having an IP.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 26, 2014 at 3:49 PM, Lakmal Warusawithana <
>>>>>>>>>>>> lakmal@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Shall we used metadata service to publish all these generic
>>>>>>>>>>>>> information?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 3:40 PM, Imesh Gunaratne <
>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> As I can see in the agent.conf some configuration parameters
>>>>>>>>>>>>>> are missing:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> These parameters are not sent in Kuberbenetes workflow, we
>>>>>>>>>>>>>> might need to think how to send them. ASFAIK in VM workflow most of these
>>>>>>>>>>>>>> missing parameters are sent by puppet master.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 3:18 PM, Imesh Gunaratne <
>>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Seems like we have a string formatting issue in agent:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://stackoverflow.com/questions/18053500/typeerror-not-all-arguments-converted-during-string-formatting-python
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 3:11 PM, Imesh Gunaratne <
>>>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I fixed the above issue with INTERNAL parameter and now I'm
>>>>>>>>>>>>>>>> seeing the following error:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [2014-12-26 09:37:27,321] INFO
>>>>>>>>>>>>>>>> {cartridgeagentconfiguration.py:__init__} - Cartridge agent configuration
>>>>>>>>>>>>>>>> initialized
>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 850,
>>>>>>>>>>>>>>>> in emit
>>>>>>>>>>>>>>>>     msg = self.format(record)
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 723,
>>>>>>>>>>>>>>>> in format
>>>>>>>>>>>>>>>>     return fmt.format(record)
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 464,
>>>>>>>>>>>>>>>> in format
>>>>>>>>>>>>>>>>     record.message = record.getMessage()
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 328,
>>>>>>>>>>>>>>>> in getMessage
>>>>>>>>>>>>>>>>     msg = msg % self.args
>>>>>>>>>>>>>>>> TypeError: not all arguments converted during string
>>>>>>>>>>>>>>>> formatting
>>>>>>>>>>>>>>>> Logged from file agent.py, line 331
>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 850,
>>>>>>>>>>>>>>>> in emit
>>>>>>>>>>>>>>>>     msg = self.format(record)
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 723,
>>>>>>>>>>>>>>>> in format
>>>>>>>>>>>>>>>>     return fmt.format(record)
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 464,
>>>>>>>>>>>>>>>> in format
>>>>>>>>>>>>>>>>     record.message = record.getMessage()
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 328,
>>>>>>>>>>>>>>>> in getMessage
>>>>>>>>>>>>>>>>     msg = msg % self.args
>>>>>>>>>>>>>>>> TypeError: not all arguments converted during string
>>>>>>>>>>>>>>>> formatting
>>>>>>>>>>>>>>>> Logged from file agent.py, line 331
>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 850,
>>>>>>>>>>>>>>>> in emit
>>>>>>>>>>>>>>>>     msg = self.format(record)
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 723,
>>>>>>>>>>>>>>>> in format
>>>>>>>>>>>>>>>>     return fmt.format(record)
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 464,
>>>>>>>>>>>>>>>> in format
>>>>>>>>>>>>>>>>     record.message = record.getMessage()
>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 328,
>>>>>>>>>>>>>>>> in getMessage
>>>>>>>>>>>>>>>>     msg = msg % self.args
>>>>>>>>>>>>>>>> TypeError: not all arguments converted during string
>>>>>>>>>>>>>>>> formatting
>>>>>>>>>>>>>>>> Logged from file agent.py, line 331
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 3:03 PM, Imesh Gunaratne <
>>>>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As I found there was an issue in agent code, it was using
>>>>>>>>>>>>>>>>> PROVIDER constant to fetch the INTERNAL parameter. I cannot see PROVIDER
>>>>>>>>>>>>>>>>> parameter being using in the agent.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 2:32 PM, Gayan Gunarathne <
>>>>>>>>>>>>>>>>> gayang@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Imesh,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Attachment is the sample launch_params that I have tested.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Gayan
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 2:22 PM, Gayan Gunarathne <
>>>>>>>>>>>>>>>>>> gayang@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Imesh,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is that launch_params contains the PROVIDER parameter? I
>>>>>>>>>>>>>>>>>>> guess this comes when the provider is not available in the payload. Seems
>>>>>>>>>>>>>>>>>>> like we are showing wrong error message.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Gayan
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 2:05 PM, Imesh Gunaratne <
>>>>>>>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks Gayan, yes it was not there in launch_params, I
>>>>>>>>>>>>>>>>>>>> just fixed it and seeing the below error:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [2014-12-26 08:31:17,311] INFO
>>>>>>>>>>>>>>>>>>>> {cartridgeagentconfiguration.py:__init__} -  INTERNAL payload parameter is
>>>>>>>>>>>>>>>>>>>> not found
>>>>>>>>>>>>>>>>>>>> [2014-12-26 08:31:17,311] INFO
>>>>>>>>>>>>>>>>>>>> {cartridgeagentconfiguration.py:__init__} - Cartridge agent configuration
>>>>>>>>>>>>>>>>>>>> initialized
>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line
>>>>>>>>>>>>>>>>>>>> 850, in emit
>>>>>>>>>>>>>>>>>>>>     msg = self.format(record)
>>>>>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line
>>>>>>>>>>>>>>>>>>>> 723, in format
>>>>>>>>>>>>>>>>>>>>     return fmt.format(record)
>>>>>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line
>>>>>>>>>>>>>>>>>>>> 464, in format
>>>>>>>>>>>>>>>>>>>>     record.message = record.getMessage()
>>>>>>>>>>>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line
>>>>>>>>>>>>>>>>>>>> 328, in getMessage
>>>>>>>>>>>>>>>>>>>>     msg = msg % self.args
>>>>>>>>>>>>>>>>>>>> TypeError: not all arguments converted during string
>>>>>>>>>>>>>>>>>>>> formatting
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 1:49 PM, Gayan Gunarathne <
>>>>>>>>>>>>>>>>>>>> gayang@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Imesh,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This is happen when CLUSTER_INSTANCE_ID not contain
>>>>>>>>>>>>>>>>>>>>> as the payload parameter of the cartridge agent.I think we can first check
>>>>>>>>>>>>>>>>>>>>> the agent payload and see CLUSTER_INSTANCE_ID is
>>>>>>>>>>>>>>>>>>>>> there in the agent payload.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Gayan
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 1:39 PM, Imesh Gunaratne <
>>>>>>>>>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I fixed the above issue in cartridge agent and
>>>>>>>>>>>>>>>>>>>>>> re-created the base image, now I'm seeing another error:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> root@e82ebdc6-8cd5-11e4-85d5-080027f35f1b:~# cat
>>>>>>>>>>>>>>>>>>>>>> /tmp/agent.screen.log
>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>   File "agent.py", line 32, in <module>
>>>>>>>>>>>>>>>>>>>>>>     from modules.extensions import
>>>>>>>>>>>>>>>>>>>>>> defaultextensionhandler
>>>>>>>>>>>>>>>>>>>>>>   File
>>>>>>>>>>>>>>>>>>>>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/extensions/defaultextensionhandler.py",
>>>>>>>>>>>>>>>>>>>>>> line 22, in <module>
>>>>>>>>>>>>>>>>>>>>>>     from ..util import extensionutils,
>>>>>>>>>>>>>>>>>>>>>> cartridgeagentutils
>>>>>>>>>>>>>>>>>>>>>>   File
>>>>>>>>>>>>>>>>>>>>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/util/extensionutils.py",
>>>>>>>>>>>>>>>>>>>>>> line 29, in <module>
>>>>>>>>>>>>>>>>>>>>>>     cartridge_agent_config =
>>>>>>>>>>>>>>>>>>>>>> cartridgeagentconfiguration.CartridgeAgentConfiguration()
>>>>>>>>>>>>>>>>>>>>>>   File
>>>>>>>>>>>>>>>>>>>>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/config/cartridgeagentconfiguration.py",
>>>>>>>>>>>>>>>>>>>>>> line 349, in __init__
>>>>>>>>>>>>>>>>>>>>>>     CartridgeAgentConfiguration.instance =
>>>>>>>>>>>>>>>>>>>>>> CartridgeAgentConfiguration.__CartridgeAgentConfiguration()
>>>>>>>>>>>>>>>>>>>>>>   File
>>>>>>>>>>>>>>>>>>>>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/config/cartridgeagentconfiguration.py",
>>>>>>>>>>>>>>>>>>>>>> line 226, in __init__
>>>>>>>>>>>>>>>>>>>>>>     raise RuntimeError(ex)
>>>>>>>>>>>>>>>>>>>>>> RuntimeError: Cannot find the value of required
>>>>>>>>>>>>>>>>>>>>>> parameter: 'CLUSTER_INSTANCE_ID'
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I can see CLUSTER_INSTANCE_ID being passed to the
>>>>>>>>>>>>>>>>>>>>>> container, may be there is something wrong in the agent code, will
>>>>>>>>>>>>>>>>>>>>>> investigate.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 26, 2014 at 2:40 AM, Imesh Gunaratne <
>>>>>>>>>>>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Devs,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I fixed some of the issues in Kubernetes Iaas and
>>>>>>>>>>>>>>>>>>>>>>> tested the Kubernetes workflow with sample php docker image. Now
>>>>>>>>>>>>>>>>>>>>>>> replication controllers, services and containers get created without a
>>>>>>>>>>>>>>>>>>>>>>> problem. We can also access the php server in the container.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> However I noticed that agent does not send the
>>>>>>>>>>>>>>>>>>>>>>> instance started event to CEP. Following error was found in agent screen
>>>>>>>>>>>>>>>>>>>>>>> log:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>   File "agent.py", line 21, in <module>
>>>>>>>>>>>>>>>>>>>>>>>     from cartridgeagent.cartridgeagent.modules
>>>>>>>>>>>>>>>>>>>>>>> import eventsubscriber
>>>>>>>>>>>>>>>>>>>>>>> ImportError: No module named
>>>>>>>>>>>>>>>>>>>>>>> cartridgeagent.cartridgeagent.modules
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> May be agent was not updated in the php docker
>>>>>>>>>>>>>>>>>>>>>>> image. Will check this tomorrow.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>>>>>>>>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>>>>>>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>>>>>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Lakmal Warusawithana
>>>>>>>>>>>>> Vice President, Apache Stratos
>>>>>>>>>>>>> Director - Cloud Architecture; WSO2 Inc.
>>>>>>>>>>>>> Mobile : +94714289692
>>>>>>>>>>>>> Blog : http://lakmalsview.blogspot.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>
>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Rajkumar Rajaratnam
>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>> Software Engineer, WSO2
>>>>>>>>>>>
>>>>>>>>>>> Mobile : +94777568639
>>>>>>>>>>> Blog : rajkumarr.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>
>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Imesh Gunaratne
>>>>>>>>>
>>>>>>>>> Technical Lead, WSO2
>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Imesh Gunaratne
>>>>>>>>
>>>>>>>> Technical Lead, WSO2
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Lakmal Warusawithana
>>>>>>> Vice President, Apache Stratos
>>>>>>> Director - Cloud Architecture; WSO2 Inc.
>>>>>>> Mobile : +94714289692
>>>>>>> Blog : http://lakmalsview.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Rajkumar Rajaratnam
>>>> Committer & PMC Member, Apache Stratos
>>>> Software Engineer, WSO2
>>>>
>>>> Mobile : +94777568639
>>>> Blog : rajkumarr.com
>>>>
>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>
>
> --
> Rajkumar Rajaratnam
> Committer & PMC Member, Apache Stratos
> Software Engineer, WSO2
>
> Mobile : +94777568639
> Blog : rajkumarr.com
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos