You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Keith Wall (JIRA)" <ji...@apache.org> on 2016/11/24 16:48:58 UTC

[jira] [Commented] (QPID-7425) Management delete of a message may refund credit too early

    [ https://issues.apache.org/jira/browse/QPID-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693736#comment-15693736 ] 

Keith Wall commented on QPID-7425:
----------------------------------

Since the elimination of queue runner, this change becomes required. We said that we need to delay the refund of credit on transactional sessions until the session commits.  For non-transactional sessions the ack needs to restore the credit. The credit needs to be restored even if the message has been stolen from us and deleted.

The following stack is currently possible on trunk:


{noformat}
2016-11-23 10:28:50,235         ERROR [HttpManagement-http-290417] o.a.q.s.m.p.f.ExceptionHandlingFilter Unexpected exception in servlet '/api/latest/queue/test/test/LiveQueueOperationsTest-testClearQueueOperationWithActiveConsumerDlqAll/clearQueue':
java.lang.IllegalStateException: Consumer credit accounting error. Restored more credit than we ever had: messageCredit=1001  messageCreditLimit=1000
        at org.apache.qpid.server.protocol.v0_8.Pre0_10CreditManager.restoreCredit(Pre0_10CreditManager.java:73) ~[qpid-broker-plugins-amqp-0-8-protocol-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.protocol.v0_8.ConsumerTarget_0_8.restoreCredit(ConsumerTarget_0_8.java:392) ~[qpid-broker-plugins-amqp-0-8-protocol-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.protocol.v0_8.ConsumerTarget_0_8.removeUnacknowledgedMessage(ConsumerTarget_0_8.java:458) ~[qpid-broker-plugins-amqp-0-8-protocol-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.protocol.v0_8.ConsumerTarget_0_8.access$000(ConsumerTarget_0_8.java:47) ~[qpid-broker-plugins-amqp-0-8-protocol-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.protocol.v0_8.ConsumerTarget_0_8$1.stateChanged(ConsumerTarget_0_8.java:483) ~[qpid-broker-plugins-amqp-0-8-protocol-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.protocol.v0_8.ConsumerTarget_0_8$1.stateChanged(ConsumerTarget_0_8.java:477) ~[qpid-broker-plugins-amqp-0-8-protocol-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.queue.QueueEntryImpl.notifyStateChange(QueueEntryImpl.java:540) ~[qpid-broker-core-6.2.0-SNAPSHOT.jar:na]
        at org.apache.qpid.server.queue.QueueEntryImpl.removeAcquisitionFromConsumer(QueueEntryImpl.java:391) ~[qpid-broker-core-6.2.0-SNAPSHOT.jar:na]
        at org.apache.qpid.server.queue.QueueEntryImpl.acquireOrSteal(QueueEntryImpl.java:267) ~[qpid-broker-core-6.2.0-SNAPSHOT.jar:na]
        at org.apache.qpid.server.queue.AbstractQueue.clearQueue(AbstractQueue.java:1575) ~[qpid-broker-core-6.2.0-SNAPSHOT.jar:na]
        at org.apache.qpid.server.queue.StandardQueueImplWithAccessChecking.clearQueue(StandardQueueImplWithAccessChecking.java:101) ~[qpid-broker-core-6.2.0-SNAPSHOT.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_67]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_67]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67]
        at org.apache.qpid.server.model.ConfiguredObjectMethodOperation.perform(ConfiguredObjectMethodOperation.java:128) ~[qpid-broker-core-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.management.plugin.servlet.rest.RestServlet.doOperation(RestServlet.java:665) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.management.plugin.servlet.rest.RestServlet.performCreateOrUpdate(RestServlet.java:502) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.management.plugin.servlet.rest.RestServlet.doPostWithSubjectAndActor(RestServlet.java:1063) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.management.plugin.servlet.rest.AbstractServlet$2.run(AbstractServlet.java:121) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.management.plugin.servlet.rest.AbstractServlet$2.run(AbstractServlet.java:117) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
        at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67]
        at org.apache.qpid.server.management.plugin.servlet.rest.AbstractServlet.doWithSubjectAndActor(AbstractServlet.java:218) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.apache.qpid.server.management.plugin.servlet.rest.AbstractServlet.doPost(AbstractServlet.java:115) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) ~[geronimo-servlet_3.0_spec-1.0.jar:1.0]
        at org.apache.qpid.server.management.plugin.servlet.rest.RestServlet.service(RestServlet.java:426) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) ~[geronimo-servlet_3.0_spec-1.0.jar:1.0]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) ~[jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.apache.qpid.server.management.plugin.filter.ForbiddingAuthorisationFilter.doFilter(ForbiddingAuthorisationFilter.java:94) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.apache.qpid.server.management.plugin.filter.ForbiddingTraceFilter.doFilter(ForbiddingTraceFilter.java:65) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.apache.qpid.server.management.plugin.filter.LoggingFilter.doFilter(LoggingFilter.java:65) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.apache.qpid.server.management.plugin.filter.PreemptiveSessionInvalidationFilter.doFilter(PreemptiveSessionInvalidationFilter.java:53) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) ~[jetty-servlets-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) ~[jetty-servlets-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.apache.qpid.server.management.plugin.filter.ExceptionHandlingFilter.doFilter(ExceptionHandlingFilter.java:56) ~[qpid-broker-plugins-management-http-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) [jetty-servlet-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.Server.handle(Server.java:370) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) [jetty-http-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) [jetty-http-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) [jetty-io-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) [jetty-io-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.17.v20150415.jar:8.1.17.v20150415]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.17.v20150415.jar:8.1.17.v20150415]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
{noformat}



> Management delete of a message may refund credit too early
> ----------------------------------------------------------
>
>                 Key: QPID-7425
>                 URL: https://issues.apache.org/jira/browse/QPID-7425
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>            Reporter: Keith Wall
>             Fix For: qpid-java-6.2
>
>
> As an operator, I am able to delete messages via Management.  Messages can be deleted even though they have already been sent to a consumer.    In this case, the message is still delete and if the client later acks or rejects a message, the ack/reject is ignored, as the message is already gone.
> However, internally the Broker restores credit to the consumer at the point in time the message is deleted.  This may allow more messages to the flow to the consumer, possibly exhausting the client's memory.  This is most likely to happen on queues with considerable depth.
> In the case where a messages is delete that has already been sent a consumer, restore of the credit should be delayed until the client acks/rejects the message.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org