[GitHub] [qpid-dispatch] kgiusti opened a new pull request #716: DISPATCH-1545: Prevent Head-of-line blocking on shared inter-router l…

[GitBox <gi...@apache.org>]

kgiusti opened a new pull request #716: DISPATCH-1545: Prevent Head-of-line blocking on shared inter-router l…
URL: https://github.com/apache/qpid-dispatch/pull/716
 
 
   …inks
   
   This patch prevents head-of-line blocking on shared links by
   dynamically assigning a dedicated inter-router link for the transfer
   of a streaming message.
   
   **Patch Notes**
   
   There are 3 primary changes made by this patch:
   
   - Classification of inbound messages as streaming
   - Modifications to the forwarding path to handle streaming messages
   - The creation of a per-connection pool of links available for streaming a message
   
   
   **Classification**
   
   There is no practical way to determine the length of an inbound message prior to its complete arrival.  This means the router has to monitor the size of an incoming message and classify it as streaming once it gets "too big" (and has not yet finished arriving).  This requires defining a size threshold over which the message is classified as streaming.
   
   Note well that this classification MUST happen BEFORE the message is forwarded by the core: the core uses the classification to determine the proper outgoing link.
   
   This means the I/O thread must buffer enough of an incoming message to make the classification before the message can be handed to the core for forwarding.  In other words an incoming message will be buffered until it completely arrives OR hits the "too big" threshold.
   
   Rather than introducing yet another user-facing "streaming message threshold" configuration knob (and the additional code complexity, test matrix dimension and user documentation it would require), this patch uses the existing Q2 threshold to classify the message. The Q2 threshold makes sense: it already sets a topmost limit for incoming message buffering.  Once Q2 is hit the router MUST forward the message in order to unblock the link, so Q2 essentially forces us to make that classification in any case.
   
   The patch does this buffering/threshold check in the rx handler in router_node.c at the point in the code where the message has yet to be forwarded.  Once forwarded further changes to the Q2 state are irrelevant to the classification process.
   
   **Forwarding Path Modifications**
   
   Once the classification is complete all streaming messages are flagged.  I've done this rather than constantly checking the "qd_message_receive_complete()" throughout the message's lifecycle because the receive status can change at any moment via the I/O thread.  This opens to possibility of race conditions.
   
   Each forwarder has been updated to check the message's streaming flag during the outbound link selection process.  If the candidate link's connection supports outgoing streaming links the forwarder will obtain a dedicated outgoing anonymous link for forwarding the message.  This link is either created on demand or obtained from the connection's pool of free streaming links.
   
   **Connection Link Pool**
   
   Every connection that supports outgoing streaming links maintains a free streaming link pool.  This pool has been added to the qdr_connection_t structure. This pool is initialized empty.  When a forwarder needs a dedicated streaming link it will check the pool for an available link.  If none are available a new qdr_link_t will be created for the outgoing message.  This link is marked as "pooled" so when the message is finally delivered the qdr_link_t will be returned to the pool, ready for the next streaming message.
   
   The pool is designed to keep at least 128 standby links.  More links can be allocated, but links in excess of the 128 will eventually be released.  This is done by the streaming_link_scrubber core module added by this patch.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org