proton messenger, link credit allocation, and starvation.

[Ken Giusti <kg...@redhat.com>]

Hi,

While working on https://issues.apache.org/jira/browse/PROTON-200, I've been able to "hang" a messenger server by doing the following simple test:

In one thread I create a message receiver by subscribing to to "amqp://~0.0.0.0:12345", and simply consuming messages like so:

     while True:
        server.recv(10)
        while server.incoming:
           server.get(msg)
           ....

In another thread, I produce a couple of messages, each with a different "path" given to the message's address. For example:


    msg = Message()
    msg.address="amqp://0.0.0.0:12345/XXX"
    ....
    client.put(msg)
    client.send()
    sleep(1)

    msg = Message()
    msg.address="amqp://0.0.0.0:12345/YYY"
    ....
    client.put(msg)
    client.send()
    sleep(1)

    msg = Message()
    msg.address="amqp://0.0.0.0:12345/ZZZ"
    ....
    client.put(msg)
    client.send()

This ends up causing the server loop to hang in send.recv() after receiving the first message (to amqp://0.0.0.0:12345/XXX).  No other messages are retrieved. 

The problem is related to PROTON-200, but goes a bit further.  It turns out that for each client message that contains a different path in its address, a new amqp 1.0 link is created "under the covers" by messenger.  And this is fine.  However, each new link requires the receiver (server) to grant it some credit before a message can be transferred over that link.

Credit is provided by the server application via the "server.recv(10)" call - this call allows up to 10 messages to be received by granting 10 credits.  The granted credits are evenly distributed across all _known_ links, which seems like the correct thing to do.

However, the problem occurs when new links are created after the credit has been doled out.  From the example above, when server.recv(10) is invoked, only the first link has been created.  Thus, it gets all the credit.  This prevents later links from being granted any credit.  So in the example, the first link consumes all the credit, doesn't send any more messages, and starves the remaining links.

I think that this particular example could be solved by having messenger re-balance credits when new links are created - perhaps by issuing drain as described in the spec.  However this isn't a complete solution.  Even with clever credit re-balancing, if the number of links become greater than the available credit, a messenger server could hang.

So how does the application determine the proper amount of credit to provide, given that it doesn't have any control over the link population?

Or, am I just doing it wrong :) ?

-K


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org