[jira] [Commented] (DISPATCH-1487) Improve the parsing of message annotations

["Ken Giusti (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DISPATCH-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418620#comment-17418620 ] 

Ken Giusti commented on DISPATCH-1487:
--------------------------------------

Analysis of CPU hot spots in router message annotation processing (based on the fix to [DISPATCH-2251|https://issues.apache.org/jira/browse/DISPATCH-2251])

There are two general phases of processing per-message router message annotations: receive processing and transmit processing.

On message receive the router invokes qd_message_message_annotations().  This routine parses the raw message annotations into parsed fields stored in the qd_message_content_t structure.  Note: this structure is a per-message structure that represents the actual message data - it is referenced by the incoming qd_message_t and all outgoing qd_message_t per-delivery structures.  The parse fields include the following MA fields:  ingress, phase, to-override, trace, streaming-flag.  Any user MAs are also stored in the message content instance.   After these values are parsed into the message content, the router invokes router_annotate_message().  This routine performs two functions: it copies out the trace, to-override, phase, and ingress values from the content into qd_buffer_t lists attached to the qd_message_t structure.  It then computes the link-exclusions bitmask, distance, and ingress-index which are used during the routing phase.   When the routing process creates an outgoing qd_message_t the MA buffer chains are copied (cloned) from the inbound qd_message_t.

When sending out the message the router invokes compose_message_annotations if the router-specific MAs are NOT to be stripped out (like in the case of the inter-router links).  This routine gathers the MA buffer lists, etc from the qd_message_t and writes them to proton (via pn_link_send).

Using perf to analyze code behavior in a two-hop router benchmark (one producing client on the inbound router forwarding to the one consumer on the outbound router 100 byte payload messages), the number of samples taken during the above routines varies based on the role of the router - upstream (connected to the producing client) vs. downstream (connected to the consuming client).

The following illustrates the % of samples taken of the worker thread during each of the principal MA processing routines for a 15 second run of the benchmark:

 

upstream router:

0.5% in qd_message_message_annotations()

3.3% in router_annotate_message()

4.7% in compose_message_annotations()

Sum: 8.7% worker thread samples used for processing MA

downstream router:

5.9% qd_message_message_annotations()

3.2% router_annotate_message()

0.2% compose_message_annotations()

Sum: 9.3% worker thread samples used for processing MA

 

Observations:  First, in general the downstream router needs slightly more CPU resources for dealing with MA than the upstream.  The hotspots vary depending on load (annotation parsing vs annotation writing).  Assuming router_annotate_message is roughly constant regardless of mode, one can extrapolate a "middle" router on a three hop configuration would experience hot spots on both phases.

 

> Improve the parsing of message annotations
> ------------------------------------------
>
>                 Key: DISPATCH-1487
>                 URL: https://issues.apache.org/jira/browse/DISPATCH-1487
>             Project: Qpid Dispatch
>          Issue Type: Improvement
>          Components: Router Node
>    Affects Versions: 1.9.0
>            Reporter: Ken Giusti
>            Assignee: Ken Giusti
>            Priority: Major
>             Fix For: 1.18.0
>
>
> ToDo: Refactor inbound MA parsing on inbound inter-router links to improve throughput and reduce latency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org