You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@stratos.apache.org by Udara Liyanage <ud...@wso2.com> on 2015/05/07 09:07:45 UTC

Registry issue when persisting topology

Hi,

I frequently get this exception when running Stratos for longtime (10+
minutes). Please note that most failures occurs when metadata service also
save properties in the registry (not all the time, this occurs when
metadata is not persisting too).


2015-05-07 07:03:11,725] ERROR
{org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
to persist resource in registry: /cloud.controller/topology*
org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to check
the existence of the resource
/_system/governance/cloud.controller/topology. Timeout trying to lock table
"REG_RESOURCE"; SQL statement:
SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=? AND
REG_TENANT_ID=? [50200-140]
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
at
org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
at
org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
at
org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
at
org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
at
org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
at
org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
at java.util.Observable.notifyObservers(Observable.java:159)
at
org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
at
org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
at
org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
at
org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
at
org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
"REG_RESOURCE"; SQL statement:
SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=? AND
REG_TENANT_ID=? [50200-140]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.table.RegularTable.doLock(RegularTable.java:466)
at org.h2.table.RegularTable.lock(RegularTable.java:404)
at org.h2.table.TableFilter.lock(TableFilter.java:139)
at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
at org.h2.command.dml.Query.query(Query.java:241)
at org.h2.command.CommandContainer.query(CommandContainer.java:80)
at org.h2.command.Command.executeQuery(Command.java:132)
at
org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
... 23 more
[2015-05-07 07:03:11,726] FATAL
{org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
persist the Topology in registry.
org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
persist resource in registry: /cloud.controller/topology
at
org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
at
org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
at
org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
at
org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
at java.util.Observable.notifyObservers(Observable.java:159)
at
org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
at
org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
at
org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
at
org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
at
org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.wso2.carbon.registry.core.exceptions.RegistryException:
Failed to check the existence of the resource
/_system/governance/cloud.controller/topology. Timeout trying to lock table
"REG_RESOURCE"; SQL statement:
SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=? AND
REG_TENANT_ID=? [50200-140]
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
at
org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
at
org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
at
org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
... 16 more
Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
"REG_RESOURCE"; SQL statement:
SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=? AND
REG_TENANT_ID=? [50200-140]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.table.RegularTable.doLock(RegularTable.java:466)
at org.h2.table.RegularTable.lock(RegularTable.java:404)
at org.h2.table.TableFilter.lock(TableFilter.java:139)
at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
at org.h2.command.dml.Query.query(Query.java:241)
at org.h2.command.CommandContainer.query(CommandContainer.java:80)
at org.h2.command.Command.executeQuery(Command.java:132)
at
org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
... 23 more


-- 

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897

Re: Registry issue when persisting topology

Posted by Akila Ravihansa Perera <ra...@wso2.com>.

Hi Udara,

Were you able to resolve this? Looks like Stratos integration tests are
failing due to same issue. It seems Carbon registry transactions are
failing when mock iaas component is trying to persist data.

Thanks.

On Mon, May 11, 2015 at 5:34 PM, Gayan Gunarathne <ga...@wso2.com> wrote:

>
>
> Please refer the document[1]
>
> [1]
> http://dev.mysql.com/doc/innodb/1.1/en/innodb-other-changes-innodb_lock_wait_timeout.html
>
> Thanks,
> Gayan
>
> On Mon, May 11, 2015 at 5:21 PM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi Gayan,
>>
>> Nope I haven't tries that. Could you please point me to the configuration
>> to be modified?
>>
>>
>> On Mon, May 11, 2015 at 5:03 PM, Gayan Gunarathne <ga...@wso2.com>
>> wrote:
>>
>>> [2015-05-11 09:37:43,396] ERROR
>>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
>>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>>> transaction*
>>> java.sql.SQLException: Lock wait timeout exceeded; try restarting
>>> transaction
>>>
>>> What is the current lock wait timeout? Did you try with increasing this
>>> timeout?
>>>
>>> Thanks,
>>> Gayan
>>>
>>> On Mon, May 11, 2015 at 3:22 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> With MySQL, however frequency of getting the exception is lower.
>>>> If I restart Stratos at this point, the error does not exist.
>>>>
>>>> On Mon, May 11, 2015 at 3:11 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>
>>>>> Hi Isuru,
>>>>>
>>>>> Even with MySQL, similar issue seems to exist
>>>>>
>>>>> in started successfully
>>>>> [2015-05-11 09:36:43,786]  INFO
>>>>> {org.apache.stratos.autoscaler.client.CloudControllerClient} -  Trying to
>>>>> spawn an instance via cloud controller: [cluster]
>>>>> single-cartridge-app.my-as-manager.appserver-manager.domain [partition]
>>>>> partition-1 [network-partition-id] network-partition-1
>>>>> [2015-05-11 09:37:43,396] ERROR
>>>>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed
>>>>> to delete the resource with id 386. Lock wait timeout exceeded; try
>>>>> restarting transaction*
>>>>> java.sql.SQLException: Lock wait timeout exceeded; try restarting
>>>>> transaction
>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
>>>>> at
>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
>>>>> at
>>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
>>>>> at
>>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
>>>>> at
>>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1555)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
>>>>> at
>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>> at
>>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>>> at
>>>>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
>>>>> at
>>>>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
>>>>> at
>>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>>> at
>>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>>> at
>>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>>> at
>>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>> at
>>>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>>> at
>>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>>>> at
>>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>> at
>>>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>>>> at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>>> at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>> at
>>>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>>>> at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>>> at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>>> at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>>> at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>>> at
>>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>>> at
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>> at
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>>> at
>>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>>> at
>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>>> at
>>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>>> at
>>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>>> at
>>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>>> at
>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>>> at
>>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>>> at
>>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>>> at
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>> at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>>> at
>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>>> at
>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> [2015-05-11 09:37:43,397] ERROR
>>>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} -  Failed to
>>>>> persist resource in registry: /cloud.controller/topology
>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>>>>> transaction
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1563)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
>>>>> at
>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>> at
>>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>>> at org.apache.axis2.rpc.receivers.RPCMessageRecei
>>>>>
>>>>> On Fri, May 8, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> This seems to occur when metadata service too save properties in
>>>>>> registry. This occurs when multiple clients(cartridge agent) call metadata
>>>>>> service to publish property values. I did not see this error when only one
>>>>>> client talks to metadata service.
>>>>>> I guess h2 can not handle this. I will try with MySQL as Isuruh
>>>>>> mentioned since this is blocking for me now.
>>>>>>
>>>>>> On Thu, May 7, 2015 at 5:12 PM, Isuru Haththotuwa <is...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> This seems to be an issue with H2. There are many occasions that
>>>>>>> this error has been reported when using an H2 DB. Can you try:
>>>>>>>
>>>>>>> 1. replacing the currently used DB with a new H2 (from a fresh pack)
>>>>>>> 2. use a MySQL DB
>>>>>>>
>>>>>>> and see if the error is reproducible?
>>>>>>>
>>>>>>> On Thu, May 7, 2015 at 4:42 PM, Imesh Gunaratne <im...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> It looks like a database table lock is not being released until it
>>>>>>>> reaches the timeout:
>>>>>>>>
>>>>>>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>>
>>>>>>>> The concern is we do not handle any database locks in Stratos code,
>>>>>>>> they are handled by the Carbon Registry. Shall we try to narrow down the
>>>>>>>> problem by running it again without using the Metadata API?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I frequently get this exception when running Stratos for longtime
>>>>>>>>> (10+ minutes). Please note that most failures occurs when metadata service
>>>>>>>>> also save properties in the registry (not all the time, this occurs when
>>>>>>>>> metadata is not persisting too).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-05-07 07:03:11,725] ERROR
>>>>>>>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
>>>>>>>>> to persist resource in registry: /cloud.controller/topology*
>>>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed
>>>>>>>>> to check the existence of the resource
>>>>>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock
>>>>>>>>> table "REG_RESOURCE"; SQL statement:
>>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>>> at
>>>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>>>>> at
>>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>>>>> ... 23 more
>>>>>>>>> [2015-05-07 07:03:11,726] FATAL
>>>>>>>>> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
>>>>>>>>> persist the Topology in registry.
>>>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed
>>>>>>>>> to persist resource in registry: /cloud.controller/topology
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>> Caused by:
>>>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to check
>>>>>>>>> the existence of the resource
>>>>>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>>>>>> at
>>>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>>>>>> ... 16 more
>>>>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock
>>>>>>>>> table "REG_RESOURCE"; SQL statement:
>>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>>> at
>>>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>>>>> at
>>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>>>>> at
>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>>>>> ... 23 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Udara Liyanage
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2, Inc.: http://wso2.com
>>>>>>>>> lean. enterprise. middleware
>>>>>>>>>
>>>>>>>>> web: http://udaraliyanage.wordpress.com
>>>>>>>>> phone: +94 71 443 6897
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Imesh Gunaratne
>>>>>>>>
>>>>>>>> Senior Technical Lead, WSO2
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks and Regards,
>>>>>>>>
>>>>>>>> Isuru H.
>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>> * <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>> * <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Udara Liyanage
>>>>>> Software Engineer
>>>>>> WSO2, Inc.: http://wso2.com
>>>>>> lean. enterprise. middleware
>>>>>>
>>>>>> web: http://udaraliyanage.wordpress.com
>>>>>> phone: +94 71 443 6897
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Udara Liyanage
>>>>> Software Engineer
>>>>> WSO2, Inc.: http://wso2.com
>>>>> lean. enterprise. middleware
>>>>>
>>>>> web: http://udaraliyanage.wordpress.com
>>>>> phone: +94 71 443 6897
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Udara Liyanage
>>>> Software Engineer
>>>> WSO2, Inc.: http://wso2.com
>>>> lean. enterprise. middleware
>>>>
>>>> web: http://udaraliyanage.wordpress.com
>>>> phone: +94 71 443 6897
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Gayan Gunarathne
>>> Technical Lead
>>> WSO2 Inc. (http://wso2.com)
>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>
>>>
>>
>>
>>
>> --
>>
>> Udara Liyanage
>> Software Engineer
>> WSO2, Inc.: http://wso2.com
>> lean. enterprise. middleware
>>
>> web: http://udaraliyanage.wordpress.com
>> phone: +94 71 443 6897
>>
>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email  : gayang@wso2.com  | mobile : +94 766819985
>
>



-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com

Re: Registry issue when persisting topology

Posted by Gayan Gunarathne <ga...@wso2.com>.

Please refer the document[1]

[1]
http://dev.mysql.com/doc/innodb/1.1/en/innodb-other-changes-innodb_lock_wait_timeout.html

Thanks,
Gayan

On Mon, May 11, 2015 at 5:21 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Gayan,
>
> Nope I haven't tries that. Could you please point me to the configuration
> to be modified?
>
>
> On Mon, May 11, 2015 at 5:03 PM, Gayan Gunarathne <ga...@wso2.com> wrote:
>
>> [2015-05-11 09:37:43,396] ERROR
>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>> transaction*
>> java.sql.SQLException: Lock wait timeout exceeded; try restarting
>> transaction
>>
>> What is the current lock wait timeout? Did you try with increasing this
>> timeout?
>>
>> Thanks,
>> Gayan
>>
>> On Mon, May 11, 2015 at 3:22 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> With MySQL, however frequency of getting the exception is lower.
>>> If I restart Stratos at this point, the error does not exist.
>>>
>>> On Mon, May 11, 2015 at 3:11 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Hi Isuru,
>>>>
>>>> Even with MySQL, similar issue seems to exist
>>>>
>>>> in started successfully
>>>> [2015-05-11 09:36:43,786]  INFO
>>>> {org.apache.stratos.autoscaler.client.CloudControllerClient} -  Trying to
>>>> spawn an instance via cloud controller: [cluster]
>>>> single-cartridge-app.my-as-manager.appserver-manager.domain [partition]
>>>> partition-1 [network-partition-id] network-partition-1
>>>> [2015-05-11 09:37:43,396] ERROR
>>>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
>>>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>>>> transaction*
>>>> java.sql.SQLException: Lock wait timeout exceeded; try restarting
>>>> transaction
>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
>>>> at
>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
>>>> at
>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
>>>> at
>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
>>>> at
>>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1555)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
>>>> at
>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>> at
>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>> at
>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
>>>> at
>>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
>>>> at
>>>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
>>>> at
>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>> at
>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>> at
>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>> at
>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>> at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>> at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>> at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>> at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>> at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>> at
>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>> at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>> at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>> at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>> at
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>> at
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>> at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> [2015-05-11 09:37:43,397] ERROR
>>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} -  Failed to
>>>> persist resource in registry: /cloud.controller/topology
>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>>>> transaction
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1563)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
>>>> at
>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>> at
>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>> at
>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
>>>> at
>>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>> at org.apache.axis2.rpc.receivers.RPCMessageRecei
>>>>
>>>> On Fri, May 8, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> This seems to occur when metadata service too save properties in
>>>>> registry. This occurs when multiple clients(cartridge agent) call metadata
>>>>> service to publish property values. I did not see this error when only one
>>>>> client talks to metadata service.
>>>>> I guess h2 can not handle this. I will try with MySQL as Isuruh
>>>>> mentioned since this is blocking for me now.
>>>>>
>>>>> On Thu, May 7, 2015 at 5:12 PM, Isuru Haththotuwa <is...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> This seems to be an issue with H2. There are many occasions that this
>>>>>> error has been reported when using an H2 DB. Can you try:
>>>>>>
>>>>>> 1. replacing the currently used DB with a new H2 (from a fresh pack)
>>>>>> 2. use a MySQL DB
>>>>>>
>>>>>> and see if the error is reproducible?
>>>>>>
>>>>>> On Thu, May 7, 2015 at 4:42 PM, Imesh Gunaratne <im...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> It looks like a database table lock is not being released until it
>>>>>>> reaches the timeout:
>>>>>>>
>>>>>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>
>>>>>>> The concern is we do not handle any database locks in Stratos code,
>>>>>>> they are handled by the Carbon Registry. Shall we try to narrow down the
>>>>>>> problem by running it again without using the Metadata API?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>> On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I frequently get this exception when running Stratos for longtime
>>>>>>>> (10+ minutes). Please note that most failures occurs when metadata service
>>>>>>>> also save properties in the registry (not all the time, this occurs when
>>>>>>>> metadata is not persisting too).
>>>>>>>>
>>>>>>>>
>>>>>>>> 2015-05-07 07:03:11,725] ERROR
>>>>>>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
>>>>>>>> to persist resource in registry: /cloud.controller/topology*
>>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed
>>>>>>>> to check the existence of the resource
>>>>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>>>>> at
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock
>>>>>>>> table "REG_RESOURCE"; SQL statement:
>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>> at
>>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>>>> at
>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>>>> ... 23 more
>>>>>>>> [2015-05-07 07:03:11,726] FATAL
>>>>>>>> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
>>>>>>>> persist the Topology in registry.
>>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed
>>>>>>>> to persist resource in registry: /cloud.controller/topology
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>>>>> at
>>>>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>>>>> at
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>> Caused by:
>>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to check
>>>>>>>> the existence of the resource
>>>>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>>>>> at
>>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>>>>> ... 16 more
>>>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock
>>>>>>>> table "REG_RESOURCE"; SQL statement:
>>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>>> at
>>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>>>> at
>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>>>> ... 23 more
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Udara Liyanage
>>>>>>>> Software Engineer
>>>>>>>> WSO2, Inc.: http://wso2.com
>>>>>>>> lean. enterprise. middleware
>>>>>>>>
>>>>>>>> web: http://udaraliyanage.wordpress.com
>>>>>>>> phone: +94 71 443 6897
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Imesh Gunaratne
>>>>>>>
>>>>>>> Senior Technical Lead, WSO2
>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>>
>>>>>>> Isuru H.
>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>> * <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>> * <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Udara Liyanage
>>>>> Software Engineer
>>>>> WSO2, Inc.: http://wso2.com
>>>>> lean. enterprise. middleware
>>>>>
>>>>> web: http://udaraliyanage.wordpress.com
>>>>> phone: +94 71 443 6897
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Udara Liyanage
>>>> Software Engineer
>>>> WSO2, Inc.: http://wso2.com
>>>> lean. enterprise. middleware
>>>>
>>>> web: http://udaraliyanage.wordpress.com
>>>> phone: +94 71 443 6897
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Udara Liyanage
>>> Software Engineer
>>> WSO2, Inc.: http://wso2.com
>>> lean. enterprise. middleware
>>>
>>> web: http://udaraliyanage.wordpress.com
>>> phone: +94 71 443 6897
>>>
>>
>>
>>
>> --
>>
>> Gayan Gunarathne
>> Technical Lead
>> WSO2 Inc. (http://wso2.com)
>> email  : gayang@wso2.com  | mobile : +94 766819985
>>
>>
>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 

Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email  : gayang@wso2.com  | mobile : +94 766819985

Re: Registry issue when persisting topology

Posted by Udara Liyanage <ud...@wso2.com>.

Hi Gayan,

Nope I haven't tries that. Could you please point me to the configuration
to be modified?


On Mon, May 11, 2015 at 5:03 PM, Gayan Gunarathne <ga...@wso2.com> wrote:

> [2015-05-11 09:37:43,396] ERROR
> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
> delete the resource with id 386. Lock wait timeout exceeded; try restarting
> transaction*
> java.sql.SQLException: Lock wait timeout exceeded; try restarting
> transaction
>
> What is the current lock wait timeout? Did you try with increasing this
> timeout?
>
> Thanks,
> Gayan
>
> On Mon, May 11, 2015 at 3:22 PM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi,
>>
>> With MySQL, however frequency of getting the exception is lower.
>> If I restart Stratos at this point, the error does not exist.
>>
>> On Mon, May 11, 2015 at 3:11 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi Isuru,
>>>
>>> Even with MySQL, similar issue seems to exist
>>>
>>> in started successfully
>>> [2015-05-11 09:36:43,786]  INFO
>>> {org.apache.stratos.autoscaler.client.CloudControllerClient} -  Trying to
>>> spawn an instance via cloud controller: [cluster]
>>> single-cartridge-app.my-as-manager.appserver-manager.domain [partition]
>>> partition-1 [network-partition-id] network-partition-1
>>> [2015-05-11 09:37:43,396] ERROR
>>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
>>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>>> transaction*
>>> java.sql.SQLException: Lock wait timeout exceeded; try restarting
>>> transaction
>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
>>> at
>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
>>> at
>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
>>> at
>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
>>> at
>>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1555)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
>>> at org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
>>> at
>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>> at
>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>> at
>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>> at
>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
>>> at
>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
>>> at
>>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
>>> at
>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>> at
>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>> at
>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>> at
>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> at
>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>> at
>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>> at
>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>> at
>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>> at
>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>> at
>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>> at
>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>> [2015-05-11 09:37:43,397] ERROR
>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} -  Failed to
>>> persist resource in registry: /cloud.controller/topology
>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>>> transaction
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1563)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
>>> at org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
>>> at
>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>> at
>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>> at
>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>> at
>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
>>> at
>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>> at org.apache.axis2.rpc.receivers.RPCMessageRecei
>>>
>>> On Fri, May 8, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> This seems to occur when metadata service too save properties in
>>>> registry. This occurs when multiple clients(cartridge agent) call metadata
>>>> service to publish property values. I did not see this error when only one
>>>> client talks to metadata service.
>>>> I guess h2 can not handle this. I will try with MySQL as Isuruh
>>>> mentioned since this is blocking for me now.
>>>>
>>>> On Thu, May 7, 2015 at 5:12 PM, Isuru Haththotuwa <is...@apache.org>
>>>> wrote:
>>>>
>>>>> This seems to be an issue with H2. There are many occasions that this
>>>>> error has been reported when using an H2 DB. Can you try:
>>>>>
>>>>> 1. replacing the currently used DB with a new H2 (from a fresh pack)
>>>>> 2. use a MySQL DB
>>>>>
>>>>> and see if the error is reproducible?
>>>>>
>>>>> On Thu, May 7, 2015 at 4:42 PM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> It looks like a database table lock is not being released until it
>>>>>> reaches the timeout:
>>>>>>
>>>>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>
>>>>>> The concern is we do not handle any database locks in Stratos code,
>>>>>> they are handled by the Carbon Registry. Shall we try to narrow down the
>>>>>> problem by running it again without using the Metadata API?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I frequently get this exception when running Stratos for longtime
>>>>>>> (10+ minutes). Please note that most failures occurs when metadata service
>>>>>>> also save properties in the registry (not all the time, this occurs when
>>>>>>> metadata is not persisting too).
>>>>>>>
>>>>>>>
>>>>>>> 2015-05-07 07:03:11,725] ERROR
>>>>>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
>>>>>>> to persist resource in registry: /cloud.controller/topology*
>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed
>>>>>>> to check the existence of the resource
>>>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock
>>>>>>> table "REG_RESOURCE"; SQL statement:
>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>> at
>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>>> at
>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>>> ... 23 more
>>>>>>> [2015-05-07 07:03:11,726] FATAL
>>>>>>> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
>>>>>>> persist the Topology in registry.
>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed
>>>>>>> to persist resource in registry: /cloud.controller/topology
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>>>> at
>>>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>> Caused by:
>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to check
>>>>>>> the existence of the resource
>>>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>>>> at
>>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>>>> ... 16 more
>>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock
>>>>>>> table "REG_RESOURCE"; SQL statement:
>>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>>> at
>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>>> at
>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>>> ... 23 more
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Udara Liyanage
>>>>>>> Software Engineer
>>>>>>> WSO2, Inc.: http://wso2.com
>>>>>>> lean. enterprise. middleware
>>>>>>>
>>>>>>> web: http://udaraliyanage.wordpress.com
>>>>>>> phone: +94 71 443 6897
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Senior Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Isuru H.
>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>> * <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>> * <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Udara Liyanage
>>>> Software Engineer
>>>> WSO2, Inc.: http://wso2.com
>>>> lean. enterprise. middleware
>>>>
>>>> web: http://udaraliyanage.wordpress.com
>>>> phone: +94 71 443 6897
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Udara Liyanage
>>> Software Engineer
>>> WSO2, Inc.: http://wso2.com
>>> lean. enterprise. middleware
>>>
>>> web: http://udaraliyanage.wordpress.com
>>> phone: +94 71 443 6897
>>>
>>
>>
>>
>> --
>>
>> Udara Liyanage
>> Software Engineer
>> WSO2, Inc.: http://wso2.com
>> lean. enterprise. middleware
>>
>> web: http://udaraliyanage.wordpress.com
>> phone: +94 71 443 6897
>>
>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email  : gayang@wso2.com  | mobile : +94 766819985
>
>



-- 

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897

Re: Registry issue when persisting topology

Posted by Gayan Gunarathne <ga...@wso2.com>.

[2015-05-11 09:37:43,396] ERROR
{org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
delete the resource with id 386. Lock wait timeout exceeded; try restarting
transaction*
java.sql.SQLException: Lock wait timeout exceeded; try restarting
transaction

What is the current lock wait timeout? Did you try with increasing this
timeout?

Thanks,
Gayan

On Mon, May 11, 2015 at 3:22 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi,
>
> With MySQL, however frequency of getting the exception is lower.
> If I restart Stratos at this point, the error does not exist.
>
> On Mon, May 11, 2015 at 3:11 PM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi Isuru,
>>
>> Even with MySQL, similar issue seems to exist
>>
>> in started successfully
>> [2015-05-11 09:36:43,786]  INFO
>> {org.apache.stratos.autoscaler.client.CloudControllerClient} -  Trying to
>> spawn an instance via cloud controller: [cluster]
>> single-cartridge-app.my-as-manager.appserver-manager.domain [partition]
>> partition-1 [network-partition-id] network-partition-1
>> [2015-05-11 09:37:43,396] ERROR
>> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>> transaction*
>> java.sql.SQLException: Lock wait timeout exceeded; try restarting
>> transaction
>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
>> at
>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
>> at
>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
>> at
>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
>> at
>> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1555)
>> at
>> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
>> at
>> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
>> at org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
>> at
>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
>> at
>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>> at
>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>> at
>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>> at
>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>> at
>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>> at
>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
>> at
>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>> at
>> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
>> at
>> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
>> at
>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>> at
>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>> at
>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>> at
>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>> at
>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>> at
>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>> at
>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>> at
>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>> at
>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>> at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>> at
>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>> at
>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>> at
>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>> at
>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>> at
>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>> at
>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>> at
>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>> at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>> at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> [2015-05-11 09:37:43,397] ERROR
>> {org.apache.stratos.cloud.controller.registry.RegistryManager} -  Failed to
>> persist resource in registry: /cloud.controller/topology
>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>> delete the resource with id 386. Lock wait timeout exceeded; try restarting
>> transaction
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1563)
>> at
>> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
>> at
>> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
>> at org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
>> at
>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
>> at
>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>> at
>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>> at
>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>> at
>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>> at
>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>> at
>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
>> at
>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>> at org.apache.axis2.rpc.receivers.RPCMessageRecei
>>
>> On Fri, May 8, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> This seems to occur when metadata service too save properties in
>>> registry. This occurs when multiple clients(cartridge agent) call metadata
>>> service to publish property values. I did not see this error when only one
>>> client talks to metadata service.
>>> I guess h2 can not handle this. I will try with MySQL as Isuruh
>>> mentioned since this is blocking for me now.
>>>
>>> On Thu, May 7, 2015 at 5:12 PM, Isuru Haththotuwa <is...@apache.org>
>>> wrote:
>>>
>>>> This seems to be an issue with H2. There are many occasions that this
>>>> error has been reported when using an H2 DB. Can you try:
>>>>
>>>> 1. replacing the currently used DB with a new H2 (from a fresh pack)
>>>> 2. use a MySQL DB
>>>>
>>>> and see if the error is reproducible?
>>>>
>>>> On Thu, May 7, 2015 at 4:42 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> It looks like a database table lock is not being released until it
>>>>> reaches the timeout:
>>>>>
>>>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>
>>>>> The concern is we do not handle any database locks in Stratos code,
>>>>> they are handled by the Carbon Registry. Shall we try to narrow down the
>>>>> problem by running it again without using the Metadata API?
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I frequently get this exception when running Stratos for longtime
>>>>>> (10+ minutes). Please note that most failures occurs when metadata service
>>>>>> also save properties in the registry (not all the time, this occurs when
>>>>>> metadata is not persisting too).
>>>>>>
>>>>>>
>>>>>> 2015-05-07 07:03:11,725] ERROR
>>>>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
>>>>>> to persist resource in registry: /cloud.controller/topology*
>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>>>> check the existence of the resource
>>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>>> at
>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>> at
>>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>>> at
>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>>> at
>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>>> at
>>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>>> at
>>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>> at
>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>> at
>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>> ... 23 more
>>>>>> [2015-05-07 07:03:11,726] FATAL
>>>>>> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
>>>>>> persist the Topology in registry.
>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>>>> persist resource in registry: /cloud.controller/topology
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>>> at
>>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>>> at
>>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>>> at
>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>>> at
>>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>>> at
>>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>>> at
>>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by:
>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to check
>>>>>> the existence of the resource
>>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>>> at
>>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>>> ... 16 more
>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>>>>> "REG_RESOURCE"; SQL statement:
>>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>>> at
>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>>> at
>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>>> at
>>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>>> ... 23 more
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Udara Liyanage
>>>>>> Software Engineer
>>>>>> WSO2, Inc.: http://wso2.com
>>>>>> lean. enterprise. middleware
>>>>>>
>>>>>> web: http://udaraliyanage.wordpress.com
>>>>>> phone: +94 71 443 6897
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Senior Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>>
>>>>> Isuru H.
>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>
>>>>>
>>>>> * <http://wso2.com/>*
>>>>>
>>>>>
>>>>> * <http://wso2.com/>*
>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>>
>>> Udara Liyanage
>>> Software Engineer
>>> WSO2, Inc.: http://wso2.com
>>> lean. enterprise. middleware
>>>
>>> web: http://udaraliyanage.wordpress.com
>>> phone: +94 71 443 6897
>>>
>>
>>
>>
>> --
>>
>> Udara Liyanage
>> Software Engineer
>> WSO2, Inc.: http://wso2.com
>> lean. enterprise. middleware
>>
>> web: http://udaraliyanage.wordpress.com
>> phone: +94 71 443 6897
>>
>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 

Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email  : gayang@wso2.com  | mobile : +94 766819985

Re: Registry issue when persisting topology

Posted by Udara Liyanage <ud...@wso2.com>.

Hi,

With MySQL, however frequency of getting the exception is lower.
If I restart Stratos at this point, the error does not exist.

On Mon, May 11, 2015 at 3:11 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Isuru,
>
> Even with MySQL, similar issue seems to exist
>
> in started successfully
> [2015-05-11 09:36:43,786]  INFO
> {org.apache.stratos.autoscaler.client.CloudControllerClient} -  Trying to
> spawn an instance via cloud controller: [cluster]
> single-cartridge-app.my-as-manager.appserver-manager.domain [partition]
> partition-1 [network-partition-id] network-partition-1
> [2015-05-11 09:37:43,396] ERROR
> {org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
> delete the resource with id 386. Lock wait timeout exceeded; try restarting
> transaction*
> java.sql.SQLException: Lock wait timeout exceeded; try restarting
> transaction
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1555)
> at
> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
> at
> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
> at org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
> at
> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
> at
> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
> at
> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
> at
> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
> at
> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
> at
> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
> at
> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
> at
> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
> at
> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
> at
> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
> at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
> at
> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> at
> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
> at
> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
> at
> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> at
> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at
> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
> at
> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
> at
> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
> at
> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> [2015-05-11 09:37:43,397] ERROR
> {org.apache.stratos.cloud.controller.registry.RegistryManager} -  Failed to
> persist resource in registry: /cloud.controller/topology
> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
> delete the resource with id 386. Lock wait timeout exceeded; try restarting
> transaction
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1563)
> at
> org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
> at
> org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
> at org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
> at
> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
> at
> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
> at
> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
> at
> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
> at
> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
> at
> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
> at org.apache.axis2.rpc.receivers.RPCMessageRecei
>
> On Fri, May 8, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi,
>>
>> This seems to occur when metadata service too save properties in
>> registry. This occurs when multiple clients(cartridge agent) call metadata
>> service to publish property values. I did not see this error when only one
>> client talks to metadata service.
>> I guess h2 can not handle this. I will try with MySQL as Isuruh mentioned
>> since this is blocking for me now.
>>
>> On Thu, May 7, 2015 at 5:12 PM, Isuru Haththotuwa <is...@apache.org>
>> wrote:
>>
>>> This seems to be an issue with H2. There are many occasions that this
>>> error has been reported when using an H2 DB. Can you try:
>>>
>>> 1. replacing the currently used DB with a new H2 (from a fresh pack)
>>> 2. use a MySQL DB
>>>
>>> and see if the error is reproducible?
>>>
>>> On Thu, May 7, 2015 at 4:42 PM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> It looks like a database table lock is not being released until it
>>>> reaches the timeout:
>>>>
>>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>>> AND REG_TENANT_ID=? [50200-140]
>>>>
>>>> The concern is we do not handle any database locks in Stratos code,
>>>> they are handled by the Carbon Registry. Shall we try to narrow down the
>>>> problem by running it again without using the Metadata API?
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I frequently get this exception when running Stratos for longtime (10+
>>>>> minutes). Please note that most failures occurs when metadata service also
>>>>> save properties in the registry (not all the time, this occurs when
>>>>> metadata is not persisting too).
>>>>>
>>>>>
>>>>> 2015-05-07 07:03:11,725] ERROR
>>>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
>>>>> to persist resource in registry: /cloud.controller/topology*
>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>>> check the existence of the resource
>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>> "REG_RESOURCE"; SQL statement:
>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>> at
>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>> at
>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>> at
>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>> at
>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>> at
>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>> at
>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>> at
>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>>>> "REG_RESOURCE"; SQL statement:
>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>> at
>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>> ... 23 more
>>>>> [2015-05-07 07:03:11,726] FATAL
>>>>> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
>>>>> persist the Topology in registry.
>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>>> persist resource in registry: /cloud.controller/topology
>>>>> at
>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>>> at
>>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>>> at
>>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>>> at
>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>>> at
>>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>>> at
>>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>>> at
>>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> Caused by: org.wso2.carbon.registry.core.exceptions.RegistryException:
>>>>> Failed to check the existence of the resource
>>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>>> "REG_RESOURCE"; SQL statement:
>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>>> at
>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>>> at
>>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>>> ... 16 more
>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>>>> "REG_RESOURCE"; SQL statement:
>>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND
>>>>> REG_NAME=? AND REG_TENANT_ID=? [50200-140]
>>>>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>>> at
>>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>>> ... 23 more
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Udara Liyanage
>>>>> Software Engineer
>>>>> WSO2, Inc.: http://wso2.com
>>>>> lean. enterprise. middleware
>>>>>
>>>>> web: http://udaraliyanage.wordpress.com
>>>>> phone: +94 71 443 6897
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Senior Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> +94 716 358 048* <http://wso2.com/>*
>>>>
>>>>
>>>> * <http://wso2.com/>*
>>>>
>>>>
>>>> * <http://wso2.com/>*
>>>>
>>>>
>>>>
>>
>>
>> --
>>
>> Udara Liyanage
>> Software Engineer
>> WSO2, Inc.: http://wso2.com
>> lean. enterprise. middleware
>>
>> web: http://udaraliyanage.wordpress.com
>> phone: +94 71 443 6897
>>
>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897

Re: Registry issue when persisting topology

Posted by Udara Liyanage <ud...@wso2.com>.

Hi Isuru,

Even with MySQL, similar issue seems to exist

in started successfully
[2015-05-11 09:36:43,786]  INFO
{org.apache.stratos.autoscaler.client.CloudControllerClient} -  Trying to
spawn an instance via cloud controller: [cluster]
single-cartridge-app.my-as-manager.appserver-manager.domain [partition]
partition-1 [network-partition-id] network-partition-1
[2015-05-11 09:37:43,396] ERROR
{org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO} -  *Failed to
delete the resource with id 386. Lock wait timeout exceeded; try restarting
transaction*
java.sql.SQLException: Lock wait timeout exceeded; try restarting
transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1555)
at
org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
at
org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
at org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
at
org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
at
org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
at
org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
at
org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at
org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
at
org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
at
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
at
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2015-05-11 09:37:43,397] ERROR
{org.apache.stratos.cloud.controller.registry.RegistryManager} -  Failed to
persist resource in registry: /cloud.controller/topology
org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
delete the resource with id 386. Lock wait timeout exceeded; try restarting
transaction
at
org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.deleteResource(JDBCResourceDAO.java:1563)
at
org.wso2.carbon.registry.core.jdbc.Repository.removeResource(Repository.java:615)
at
org.wso2.carbon.registry.core.jdbc.Repository.prepareUpdate(Repository.java:362)
at org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:709)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
at
org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
at
org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
at
org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberCreatedEvent(TopologyBuilder.java:434)
at
org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:486)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCMessageRecei

On Fri, May 8, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi,
>
> This seems to occur when metadata service too save properties in registry.
> This occurs when multiple clients(cartridge agent) call metadata service to
> publish property values. I did not see this error when only one client
> talks to metadata service.
> I guess h2 can not handle this. I will try with MySQL as Isuruh mentioned
> since this is blocking for me now.
>
> On Thu, May 7, 2015 at 5:12 PM, Isuru Haththotuwa <is...@apache.org>
> wrote:
>
>> This seems to be an issue with H2. There are many occasions that this
>> error has been reported when using an H2 DB. Can you try:
>>
>> 1. replacing the currently used DB with a new H2 (from a fresh pack)
>> 2. use a MySQL DB
>>
>> and see if the error is reproducible?
>>
>> On Thu, May 7, 2015 at 4:42 PM, Imesh Gunaratne <im...@apache.org> wrote:
>>
>>> It looks like a database table lock is not being released until it
>>> reaches the timeout:
>>>
>>> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>> AND REG_TENANT_ID=? [50200-140]
>>>
>>> The concern is we do not handle any database locks in Stratos code, they
>>> are handled by the Carbon Registry. Shall we try to narrow down the problem
>>> by running it again without using the Metadata API?
>>>
>>> Thanks
>>>
>>>
>>> On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I frequently get this exception when running Stratos for longtime (10+
>>>> minutes). Please note that most failures occurs when metadata service also
>>>> save properties in the registry (not all the time, this occurs when
>>>> metadata is not persisting too).
>>>>
>>>>
>>>> 2015-05-07 07:03:11,725] ERROR
>>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
>>>> to persist resource in registry: /cloud.controller/topology*
>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>> check the existence of the resource
>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>> "REG_RESOURCE"; SQL statement:
>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>>> AND REG_TENANT_ID=? [50200-140]
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>> at
>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>> at
>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>> at
>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>> at
>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>> at
>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>> at
>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>> at
>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>> at
>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>> at
>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>>> "REG_RESOURCE"; SQL statement:
>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>>> AND REG_TENANT_ID=? [50200-140]
>>>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>> at
>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>> ... 23 more
>>>> [2015-05-07 07:03:11,726] FATAL
>>>> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
>>>> persist the Topology in registry.
>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>>> persist resource in registry: /cloud.controller/topology
>>>> at
>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
>>>> at
>>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>>> at
>>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>>> at
>>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>>> at
>>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>>> at
>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>>> at
>>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>>> at
>>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>>> at
>>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> Caused by: org.wso2.carbon.registry.core.exceptions.RegistryException:
>>>> Failed to check the existence of the resource
>>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>>> "REG_RESOURCE"; SQL statement:
>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>>> AND REG_TENANT_ID=? [50200-140]
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>>> at
>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>>> at
>>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>>> at
>>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>>> ... 16 more
>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>>> "REG_RESOURCE"; SQL statement:
>>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>>> AND REG_TENANT_ID=? [50200-140]
>>>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>>> at org.h2.message.DbException.get(DbException.java:167)
>>>> at org.h2.message.DbException.get(DbException.java:144)
>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>>> at org.h2.command.dml.Query.query(Query.java:241)
>>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>>> at
>>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>>> at
>>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>>> ... 23 more
>>>>
>>>>
>>>> --
>>>>
>>>> Udara Liyanage
>>>> Software Engineer
>>>> WSO2, Inc.: http://wso2.com
>>>> lean. enterprise. middleware
>>>>
>>>> web: http://udaraliyanage.wordpress.com
>>>> phone: +94 71 443 6897
>>>>
>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Senior Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> +94 716 358 048* <http://wso2.com/>*
>>>
>>>
>>> * <http://wso2.com/>*
>>>
>>>
>>> * <http://wso2.com/>*
>>>
>>>
>>>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>

-- 

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897

Re: Registry issue when persisting topology

Posted by Udara Liyanage <ud...@wso2.com>.

Hi,

This seems to occur when metadata service too save properties in registry.
This occurs when multiple clients(cartridge agent) call metadata service to
publish property values. I did not see this error when only one client
talks to metadata service.
I guess h2 can not handle this. I will try with MySQL as Isuruh mentioned
since this is blocking for me now.

On Thu, May 7, 2015 at 5:12 PM, Isuru Haththotuwa <is...@apache.org> wrote:

> This seems to be an issue with H2. There are many occasions that this
> error has been reported when using an H2 DB. Can you try:
>
> 1. replacing the currently used DB with a new H2 (from a fresh pack)
> 2. use a MySQL DB
>
> and see if the error is reproducible?
>
> On Thu, May 7, 2015 at 4:42 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> It looks like a database table lock is not being released until it
>> reaches the timeout:
>>
>> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>> AND REG_TENANT_ID=? [50200-140]
>>
>> The concern is we do not handle any database locks in Stratos code, they
>> are handled by the Carbon Registry. Shall we try to narrow down the problem
>> by running it again without using the Metadata API?
>>
>> Thanks
>>
>>
>> On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> I frequently get this exception when running Stratos for longtime (10+
>>> minutes). Please note that most failures occurs when metadata service also
>>> save properties in the registry (not all the time, this occurs when
>>> metadata is not persisting too).
>>>
>>>
>>> 2015-05-07 07:03:11,725] ERROR
>>> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
>>> to persist resource in registry: /cloud.controller/topology*
>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>> check the existence of the resource
>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>> "REG_RESOURCE"; SQL statement:
>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>> AND REG_TENANT_ID=? [50200-140]
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>> at
>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>> at
>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>> at
>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>> at
>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>> at
>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>> at
>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>> at
>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>> at
>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>> at
>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>> at
>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>> "REG_RESOURCE"; SQL statement:
>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>> AND REG_TENANT_ID=? [50200-140]
>>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>> at org.h2.message.DbException.get(DbException.java:167)
>>> at org.h2.message.DbException.get(DbException.java:144)
>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>> at org.h2.command.dml.Query.query(Query.java:241)
>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>> at
>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>> ... 23 more
>>> [2015-05-07 07:03:11,726] FATAL
>>> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
>>> persist the Topology in registry.
>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>>> persist resource in registry: /cloud.controller/topology
>>> at
>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
>>> at
>>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>>> at
>>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>>> at
>>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>>> at java.util.Observable.notifyObservers(Observable.java:159)
>>> at
>>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>>> at
>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>>> at
>>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>>> at
>>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>>> at
>>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>> Caused by: org.wso2.carbon.registry.core.exceptions.RegistryException:
>>> Failed to check the existence of the resource
>>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>>> "REG_RESOURCE"; SQL statement:
>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>> AND REG_TENANT_ID=? [50200-140]
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>>> at
>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>>> at
>>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>>> at
>>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>>> ... 16 more
>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>>> "REG_RESOURCE"; SQL statement:
>>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>>> AND REG_TENANT_ID=? [50200-140]
>>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>>> at org.h2.message.DbException.get(DbException.java:167)
>>> at org.h2.message.DbException.get(DbException.java:144)
>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>>> at org.h2.command.dml.Query.query(Query.java:241)
>>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>>> at org.h2.command.Command.executeQuery(Command.java:132)
>>> at
>>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>>> at
>>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>>> ... 23 more
>>>
>>>
>>> --
>>>
>>> Udara Liyanage
>>> Software Engineer
>>> WSO2, Inc.: http://wso2.com
>>> lean. enterprise. middleware
>>>
>>> web: http://udaraliyanage.wordpress.com
>>> phone: +94 71 443 6897
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Senior Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048* <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>


-- 

Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897

Re: Registry issue when persisting topology

Posted by Isuru Haththotuwa <is...@apache.org>.

This seems to be an issue with H2. There are many occasions that this error
has been reported when using an H2 DB. Can you try:

1. replacing the currently used DB with a new H2 (from a fresh pack)
2. use a MySQL DB

and see if the error is reproducible?

On Thu, May 7, 2015 at 4:42 PM, Imesh Gunaratne <im...@apache.org> wrote:

> It looks like a database table lock is not being released until it reaches
> the timeout:
>
> Timeout trying to lock table "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
>
> The concern is we do not handle any database locks in Stratos code, they
> are handled by the Carbon Registry. Shall we try to narrow down the problem
> by running it again without using the Metadata API?
>
> Thanks
>
>
> On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com> wrote:
>
>> Hi,
>>
>> I frequently get this exception when running Stratos for longtime (10+
>> minutes). Please note that most failures occurs when metadata service also
>> save properties in the registry (not all the time, this occurs when
>> metadata is not persisting too).
>>
>>
>> 2015-05-07 07:03:11,725] ERROR
>> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
>> to persist resource in registry: /cloud.controller/topology*
>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>> check the existence of the resource
>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>> "REG_RESOURCE"; SQL statement:
>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>> AND REG_TENANT_ID=? [50200-140]
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>> at
>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>> at
>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>> at
>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>> at
>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>> at
>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>> at
>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>> at
>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>> at
>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>> at
>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>> at
>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>> at java.util.Observable.notifyObservers(Observable.java:159)
>> at
>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>> at
>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>> at
>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>> at
>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>> at
>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>> "REG_RESOURCE"; SQL statement:
>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>> AND REG_TENANT_ID=? [50200-140]
>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>> at org.h2.message.DbException.get(DbException.java:167)
>> at org.h2.message.DbException.get(DbException.java:144)
>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>> at org.h2.command.dml.Query.query(Query.java:241)
>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>> at org.h2.command.Command.executeQuery(Command.java:132)
>> at
>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>> ... 23 more
>> [2015-05-07 07:03:11,726] FATAL
>> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
>> persist the Topology in registry.
>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
>> persist resource in registry: /cloud.controller/topology
>> at
>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
>> at
>> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
>> at
>> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
>> at
>> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
>> at
>> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
>> at
>> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
>> at java.util.Observable.notifyObservers(Observable.java:159)
>> at
>> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
>> at
>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
>> at
>> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
>> at
>> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
>> at
>> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.wso2.carbon.registry.core.exceptions.RegistryException:
>> Failed to check the existence of the resource
>> /_system/governance/cloud.controller/topology. Timeout trying to lock table
>> "REG_RESOURCE"; SQL statement:
>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>> AND REG_TENANT_ID=? [50200-140]
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
>> at
>> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
>> at
>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
>> at
>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
>> at
>> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
>> at
>> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
>> ... 16 more
>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
>> "REG_RESOURCE"; SQL statement:
>> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
>> AND REG_TENANT_ID=? [50200-140]
>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
>> at org.h2.message.DbException.get(DbException.java:167)
>> at org.h2.message.DbException.get(DbException.java:144)
>> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
>> at org.h2.table.RegularTable.lock(RegularTable.java:404)
>> at org.h2.table.TableFilter.lock(TableFilter.java:139)
>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
>> at org.h2.command.dml.Query.query(Query.java:241)
>> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
>> at org.h2.command.Command.executeQuery(Command.java:132)
>> at
>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
>> ... 23 more
>>
>>
>> --
>>
>> Udara Liyanage
>> Software Engineer
>> WSO2, Inc.: http://wso2.com
>> lean. enterprise. middleware
>>
>> web: http://udaraliyanage.wordpress.com
>> phone: +94 71 443 6897
>>
>
>
>
> --
> Imesh Gunaratne
>
> Senior Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: Registry issue when persisting topology

Posted by Imesh Gunaratne <im...@apache.org>.

It looks like a database table lock is not being released until it reaches
the timeout:

Timeout trying to lock table "REG_RESOURCE"; SQL statement:
SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=? AND
REG_TENANT_ID=? [50200-140]

The concern is we do not handle any database locks in Stratos code, they
are handled by the Carbon Registry. Shall we try to narrow down the problem
by running it again without using the Metadata API?

Thanks


On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi,
>
> I frequently get this exception when running Stratos for longtime (10+
> minutes). Please note that most failures occurs when metadata service also
> save properties in the registry (not all the time, this occurs when
> metadata is not persisting too).
>
>
> 2015-05-07 07:03:11,725] ERROR
> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
> to persist resource in registry: /cloud.controller/topology*
> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
> check the existence of the resource
> /_system/governance/cloud.controller/topology. Timeout trying to lock table
> "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
> at
> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
> at
> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
> at
> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
> at
> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
> at
> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
> at
> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
> at
> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
> at
> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
> at java.util.Observable.notifyObservers(Observable.java:159)
> at
> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
> at
> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
> at
> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
> at
> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
> at
> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
> "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
> at org.h2.message.DbException.get(DbException.java:167)
> at org.h2.message.DbException.get(DbException.java:144)
> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
> at org.h2.table.RegularTable.lock(RegularTable.java:404)
> at org.h2.table.TableFilter.lock(TableFilter.java:139)
> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
> at org.h2.command.dml.Query.query(Query.java:241)
> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
> at org.h2.command.Command.executeQuery(Command.java:132)
> at
> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
> ... 23 more
> [2015-05-07 07:03:11,726] FATAL
> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} -  Failed to
> persist the Topology in registry.
> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
> persist resource in registry: /cloud.controller/topology
> at
> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
> at
> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
> at
> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
> at
> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
> at java.util.Observable.notifyObservers(Observable.java:159)
> at
> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
> at
> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
> at
> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
> at
> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
> at
> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.wso2.carbon.registry.core.exceptions.RegistryException:
> Failed to check the existence of the resource
> /_system/governance/cloud.controller/topology. Timeout trying to lock table
> "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
> at
> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
> at
> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
> at
> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
> at
> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
> at
> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
> ... 16 more
> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
> "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
> at org.h2.message.DbException.get(DbException.java:167)
> at org.h2.message.DbException.get(DbException.java:144)
> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
> at org.h2.table.RegularTable.lock(RegularTable.java:404)
> at org.h2.table.TableFilter.lock(TableFilter.java:139)
> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
> at org.h2.command.dml.Query.query(Query.java:241)
> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
> at org.h2.command.Command.executeQuery(Command.java:132)
> at
> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
> ... 23 more
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>



-- 
Imesh Gunaratne

Senior Technical Lead, WSO2
Committer & PMC Member, Apache Stratos