You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficserver.apache.org by zw...@apache.org on 2022/06/07 16:27:20 UTC

[trafficserver] 01/02: money_trace: allow custom header, change span-id gen, opt to create if none (#8655)

This is an automated email from the ASF dual-hosted git repository.

zwoop pushed a commit to branch 9.2.x
in repository https://gitbox.apache.org/repos/asf/trafficserver.git

commit f9dbf571bed2acfb50744322e7ed591592f87fdf
Author: Brian Olsen <br...@comcast.com>
AuthorDate: Mon Mar 21 16:47:25 2022 -0600

    money_trace: allow custom header, change span-id gen, opt to create if none (#8655)
    
    * money_trace: add global support, custom header name, create if none.
    
    This enhances the money_trace to allow it to run globally, with the
    ability to override the global with a remap plugin.
    Adds in better logging support for the current transaction's trace header.
    Allows passthru mode which passes along the client trace header
    without modification making ATS transparent.
    
    * money_trace: change logic for how an invalid incoming header is handled
    
    * money_trace autest: remove commented code
    
    (cherry picked from commit 6e12143c550695ee7b6b0f5aff4ea3309d3acfe8)
---
 doc/admin-guide/plugins/money_trace.en.rst         | 131 +++-
 plugins/experimental/money_trace/README            |  45 +-
 plugins/experimental/money_trace/money_trace.cc    | 697 ++++++++++++++-------
 plugins/experimental/money_trace/money_trace.h     |  31 -
 .../pluginTest/money_trace/gold/global-log.gold    |   3 +
 .../pluginTest/money_trace/gold/remap-log.gold     |  23 +
 .../pluginTest/money_trace/money_trace.test.py     | 225 +++++++
 .../money_trace/money_trace_global.test.py         | 126 ++++
 8 files changed, 991 insertions(+), 290 deletions(-)

diff --git a/doc/admin-guide/plugins/money_trace.en.rst b/doc/admin-guide/plugins/money_trace.en.rst
index 41d7406dc..02c63e6ae 100644
--- a/doc/admin-guide/plugins/money_trace.en.rst
+++ b/doc/admin-guide/plugins/money_trace.en.rst
@@ -18,37 +18,54 @@
 
 
 Money Trace Plugin
-==================
+******************
 
-This is a remap plugin  that allows ATS to participate in a distributed tracing system based upon
-the Comcast "Money" distributed tracing and monitoring library.  The Comcast "Money" library has
-its roots in Google's Dapper and Twitters Zipkin systems.  A money trace header or session id, is
-attached to transaction and allows an operator with the appropriate logging systems in place,
-to determine where errors and/or latency may exit.
+Description
+===========
 
-Use of the library enables the tracing of a transaction through all systems that participate in
-handling the request. See the documentation on this open source library at
-https://github.com/Comcast/money.
+This plugin allows ATS to participate in a distributed tracing system
+based upon the Comcast "Money" distributed tracing and monitoring library.
+The Comcast "Money" library has its roots in Google's Dapper and Twitters
+Zipkin systems.  A money trace header or session id, is attached to
+transaction and allows an operator with the appropriate logging systems
+in place, to determine where errors and/or latency may exit.
+
+Use of the library enables the tracing of a transaction through all
+systems that participate in handling the request. See the documentation
+on this open source library at https://github.com/Comcast/money.
 
 How it Works
-------------
-
-This plugin checks incoming requests for the "X-MoneyTrace" header.  If the header is not present
-no further processing takes place.  However if the header is present,  the plugin will check to
-to see if the request has been cached.  If so, the plugin will add the "X-Moneytrace" header from the
-incoming request to the cached response returned to the client as required by the money_trace
-protocol.  If the request has not been cached, the plugin will extends the trace context by creating a new
-"X-MoneyTrace" header for inclusion in the outgoing request to a parent cache or origin server.
-The extended header includes the 'trace-id' from the incoming request, the incoming span-id
-becomes the outgoing parent-id and the plugin generates a new random long span id for the outgoing request.
-See the documentation at the link above for a complete description on the "X-MoneyTrace" header and how
-to use and extend it in a distributed tracing system.
+============
+
+This plugin checks incoming requests for the "X-MoneyTrace" header.
+If the header is not present no further processing takes place.
+However if the header is present,  the plugin will check to to see if the
+request has been cached.  If so, the plugin will add the "X-Moneytrace"
+header from the incoming request to the cached response returned to the
+client as required by the money_trace protocol.  If the request has not
+been cached, the plugin will extends the trace context by creating a new
+"X-MoneyTrace" header for inclusion in the outgoing request to a parent
+cache or origin server.  The extended header includes the 'trace-id'
+from the incoming request, the incoming span-id becomes the outgoing
+parent-id and the plugin generates a new span id for the
+outgoing request using the current state machine id.
+
+See the documentation at the link above for a complete description on
+the "X-MoneyTrace" header and how to use and extend it in a distributed
+tracing system.
+
+A sample money-trace header:
+
+::
+
+  X-MoneyTrace: trace-id=aa234a23-189e-4cc4-98ed-b5327b1ec231-3;parent-id=0;span-id=4303691729133364974
 
 Installation
-------------
+============
 
-The `Money Trace` plugin is a :term:`remap plugin`.  Enable it by adding
-``money_trace.so`` to your :file:`remap.config` file.  There are no options.
+The `Money Trace` plugin can be either a :term:`remap plugin` or
+:term:`global plugin`.  Enable it by adding ``money_trace.so`` to your
+:file:`remap.config` file or :file:`plugin.config`.
 
 Here is an example remap.config entry:
 
@@ -57,3 +74,69 @@ Here is an example remap.config entry:
   map http://vod.foobar.com http://origin.vod.foobar.com @plugin=money_trace.so
 
 .. _MoneyTrace:    https://github.com/Comcast/money
+
+Configuration
+=============
+
+The plugin supports the following options:
+
+* ``--create-if-none=[true|false]`` (default: ``false``)
+
+If no X-MoneyTrace header is found in the client request one will
+be manufactured using the transaction UUID as trace-id,
+the transaction state machine id as span-id and parent-id set to '0'.
+
+* ``--global-skip-header=[header name]`` (default: null/disable)
+
+This setting only applies to a :term:`global plugin` instance
+and allows remap plugin instances to override :term:`global plugin`
+behavior by disabling the :term:`global plugin`
+
+Because a :term:`global plugin` runs before any :term:`remap plugin`
+in the remap phase a pregen header may still be created by the
+:term:`global plugin` if configured to do so.
+
+The global skip check is performed during the post remap phase in order
+to allow remap plugins (like `header rewrite`) to set this skip header.
+
+It is strongly suggested to use a private ATS header (begins with '@')
+as this value.
+
+* ``--header=[header name]`` (default: ``X-MoneyTrace``)
+
+Allows the money trace header to be overridden.
+
+* ``--passthru=[true|false]`` (default: ``false``)
+
+In this mode ATS acts transparently and passes the client money trace
+header through to the parent.  It also returns this same header back to
+the client.  This option ignores the --create-if-none setting.
+
+* ``--pregen-header=[header name]`` (default: null/disable)
+
+Normally the money trace header for a transaction is only added to the
+transaction server request headers.  If this argument is supplied the
+header will be generated earlier in the transaction and added to the
+client request headers.  Use this for debug or for logging the current
+transaction's money trace header.  It is suggested to use a private
+ATS header (begins with a '@') for this value.  A :file:`logging.yaml`
+entry with pregen-header=@MoneyTrace might look like:
+
+::
+
+  %<{@MoneyTrace}cqh>
+
+Robustness
+==========
+
+This plugin tries to be robust in its parsing.  At a minimum the value
+must start with `trace-id=` set to a none empty value.
+
+If `span-id=` is found in the header value that will be used as the
+parent-id for an upstream request.  Otherwise '0' will be its value.
+
+If the incoming money trace header is invalid, it is handled based
+on the --create-if-none setting.  If create-if-none is set a new
+money trace header will be generated and used.  Otherwise the
+incoming client header value will be passed through.
+
diff --git a/plugins/experimental/money_trace/README b/plugins/experimental/money_trace/README
index 902f86019..5c32d0166 100644
--- a/plugins/experimental/money_trace/README
+++ b/plugins/experimental/money_trace/README
@@ -1,27 +1,34 @@
 
 Money trace plugin
-  This is a remap plugin  that allows ATS to participate in a distributed tracing system based upon
-  the Comcast "Money" distributed tracing and monitoring library.  The Comcast "Money" library has
-  its roots in Google's Dapper and Twitters Zipkin systems.  A money trace header or session id, is
-  attached to transaction and allows an operator with the appropriate logging systems in place,
-  to determine where errors and/or latency may exit.
+  This is a remap plugin  that allows ATS to participate in a distributed
+  tracing system based upon the Comcast "Money" distributed tracing
+  and monitoring library.  The Comcast "Money" library has its roots in
+  Google's Dapper and Twitters Zipkin systems.  A money trace header or
+  session id, is attached to transaction and allows an operator with
+  the appropriate logging systems in place, to determine where errors
+  and/or latency may exit.
 
-  Use of the library enables the tracing of a transaction through all systems that participate in
-  handling the request. See the documentation on this open source library at
-  https://github.com/Comcast/money/wiki.
+  Use of the library enables the tracing of a transaction through all
+  systems that participate in handling the request. See the documentation
+  on this open source library at https://github.com/Comcast/money/wiki.
 
-  This plugin checks incoming requests for the "X-MoneyTrace" header.  If the header is not present
-  no further processing takes place.  However if the header is present,  the plugin will check to
-  to see if the request has been cached.  If so, the plugin will add the "X-Moneytrace" header from the
-  incoming request to the cached response returned to the client as required by the money_trace
-  protocol.  If the request has not been cached, the plugin will extends the trace context by creating a new
-  "X-MoneyTrace" header for inclusion in the outgoing request to a parent cache or origin server.
-  The extended header includes the 'trace-id' from the incoming request, the incoming span-id
-  becomes the outgoing parent-id and the plugin generates a new random long span id for the outgoing request.
-  See the documentation at the link above for a complete description on the "X-MoneyTrace" header and how
-  to use and extend it in a distributed tracing system.
+  This plugin checks incoming requests for the "X-MoneyTrace" header.
+  If the header is not present no further processing takes place.
+  However if the header is present,  the plugin will check to to
+  see if the request has been cached.  If so, the plugin will add the
+  "X-Moneytrace" header from the incoming request to the cached response
+  returned to the client as required by the money_trace protocol.
+  If the request has not been cached, the plugin will extends the trace
+  context by creating a new "X-MoneyTrace" header for inclusion in the
+  outgoing request to a parent cache or origin server.  The extended
+  header includes the 'trace-id' from the incoming request, the incoming
+  span-id becomes the outgoing parent-id and the plugin generates a new
+  random long span id for the outgoing request.  See the documentation
+  at the link above for a complete description on the "X-MoneyTrace"
+  header and how to use and extend it in a distributed tracing system.
 
-  To configure and use this plugin, simply add it in the remap.config file where needed.  EXAMPLE:
+  To configure and use this plugin, simply add it in the remap.config
+  file where needed.  EXAMPLE:
 
   map http://vod.foobar.com http://origin.vod.foobar.com @plugin=money_trace.so
 
diff --git a/plugins/experimental/money_trace/money_trace.cc b/plugins/experimental/money_trace/money_trace.cc
index d8638f316..69d325268 100644
--- a/plugins/experimental/money_trace/money_trace.cc
+++ b/plugins/experimental/money_trace/money_trace.cc
@@ -17,130 +17,361 @@
  * under the License.
  */
 
-#include <iostream>
-#include <sstream>
-#include <cstdio>
-#include <cstring>
+#include "tscore/BufferWriter.h"
 #include "ts/ts.h"
 #include "ts/remap.h"
 
+#include <getopt.h>
+#include <string_view>
+#include <string>
+
 #include "money_trace.h"
 
-/**
- * Allocate transaction data structure.
- */
-static struct txndata *
-allocTransactionData()
+namespace
+{
+std::string_view const DefaultMimeHeader = "X-MoneyTrace";
+enum PluginType { REMAP, GLOBAL };
+
+struct Config {
+  std::string header;             // override header
+  std::string pregen_header;      // generate request header during remap
+  std::string global_skip_header; // skip global plugin
+  bool create_if_none = false;
+  bool passthru       = false; // transparen mode: pass any headers through
+};
+
+Config *
+config_from_args(int const argc, char const *argv[], PluginType const ptype)
 {
-  LOG_DEBUG("allocating transaction state data.");
-  struct txndata *txn_data           = static_cast<struct txndata *>(TSmalloc(sizeof(struct txndata)));
-  txn_data->client_request_mt_header = nullptr;
-  txn_data->new_span_mt_header       = nullptr;
-  return txn_data;
+  Config *const conf = new Config;
+
+  static const struct option longopt[] = {
+    {const_cast<char *>("passthru"), required_argument, nullptr, 'a'},
+    {const_cast<char *>("create-if-none"), required_argument, nullptr, 'c'},
+    {const_cast<char *>("global-skip-header"), required_argument, nullptr, 'g'},
+    {const_cast<char *>("header"), required_argument, nullptr, 'h'},
+    {const_cast<char *>("pregen-header"), required_argument, nullptr, 'p'},
+    {nullptr, 0, nullptr, 0},
+  };
+
+  // getopt assumes args start at '1' so this hack is needed
+  do {
+    int const opt = getopt_long(argc, (char *const *)argv, "a:c:h:l:p:", longopt, nullptr);
+
+    if (-1 == opt) {
+      break;
+    }
+
+    LOG_DEBUG("Opt: %c", opt);
+
+    switch (opt) {
+    case 'a':
+      if ("true" == std::string_view{optarg}) {
+        LOG_DEBUG("Plugin acts as passthrough");
+        conf->passthru = true;
+      }
+      break;
+    case 'c':
+      if ("true" == std::string_view{optarg}) {
+        LOG_DEBUG("Plugin will create header if missing");
+        conf->create_if_none = true;
+      }
+      break;
+    case 'g':
+      LOG_DEBUG("Using global-skip-header: '%s'", optarg);
+      conf->global_skip_header.assign(optarg);
+      break;
+    case 'h':
+      LOG_DEBUG("Using custom header: '%s'", optarg);
+      conf->header.assign(optarg);
+      break;
+    case 'p':
+      LOG_DEBUG("Using pregen_header '%s'", optarg);
+      conf->pregen_header.assign(optarg);
+      break;
+    default:
+      break;
+    }
+  } while (true);
+
+  if (conf->header.empty()) {
+    conf->header.assign(DefaultMimeHeader);
+    LOG_DEBUG("Using default header name: '%.*s'", (int)DefaultMimeHeader.length(), DefaultMimeHeader.data());
+  }
+
+  if (conf->passthru && conf->create_if_none) {
+    LOG_ERROR("passthru conflicts with create-if-none, disabling create-if-none!");
+    conf->create_if_none = false;
+  }
+
+  if (REMAP == ptype && !conf->global_skip_header.empty()) {
+    LOG_ERROR("--global-skip-header inappropriate for remap plugin, removing option!");
+    conf->global_skip_header.clear();
+  }
+
+  return conf;
 }
 
-/**
- * free any previously allocated transaction data.
- */
-static void
-freeTransactionData(struct txndata *txn_data)
+struct TxnData {
+  std::string client_trace;
+  std::string this_trace;
+  Config const *config = nullptr;
+};
+
+std::string_view const traceid{"trace-id="};
+std::string_view const parentid{"parent-id="};
+std::string_view const spanid{"span-id="};
+std::string_view const zerospan{"0"};
+char const sep{';'};
+
+std::string
+next_trace(std::string_view const request_hdr, TSHttpTxn const txnp)
 {
-  LOG_DEBUG("de-allocating transaction state data.");
+  std::string nexttrace;
+
+  LOG_DEBUG("next_trace with '%.*s'", (int)request_hdr.length(), request_hdr.data());
+
+  std::string_view view = request_hdr;
+
+  // trace-id must be first
+  if (0 != view.rfind(traceid, 0)) {
+    LOG_DEBUG("Expected to find prefix '%.*s' in '%.*s'", (int)traceid.length(), traceid.data(), (int)view.length(), view.data());
+    return nexttrace;
+  }
+
+  view = view.substr(traceid.length());
 
-  if (txn_data != nullptr) {
-    LOG_DEBUG("freeing transaction data.");
-    TSfree(txn_data->client_request_mt_header);
-    TSfree(txn_data->new_span_mt_header);
-    TSfree(txn_data);
+  // look for separator
+  size_t seppos = view.find_first_of(sep);
+  if (0 == seppos) {
+    LOG_DEBUG("Trace is empty for '%.*s'", (int)request_hdr.length(), request_hdr.data());
+    return nexttrace;
   }
+
+  std::string_view trace = view.substr(0, seppos);
+
+  if (std::string_view::npos == seppos) {
+    LOG_DEBUG("Expected to find separator '%c' in %.*s", sep, (int)request_hdr.length(), request_hdr.data());
+    view = std::string_view{};
+  }
+
+  std::string_view span;
+
+  // scan remaining tokens
+  while (!view.empty() && span.empty()) {
+    // skip any leading whitespace
+    while (!view.empty() && ' ' == view.front()) {
+      view = view.substr(1);
+    }
+
+    // check for span-id field
+    if (0 == view.rfind(spanid, 0)) {
+      span   = view.substr(spanid.length());
+      seppos = span.find_first_of(sep);
+      span   = span.substr(0, seppos);
+
+      // remove any trailing white space
+      while (!span.empty() && ' ' == span.back()) {
+        span = span.substr(0, span.length() - 1);
+      }
+    } else {
+      LOG_DEBUG("Non '%.*s' found in '%.*s'", (int)spanid.length(), spanid.data(), (int)view.length(), view.data());
+    }
+
+    if (span.empty()) {
+      seppos = view.find_first_of(sep);
+
+      // move forward past sep
+      if (std::string_view::npos != seppos) {
+        view = view.substr(seppos + 1);
+        LOG_DEBUG("Trimming view to '%.*s'", (int)view.length(), view.data());
+      } else {
+        view = std::string_view{};
+      }
+    }
+  }
+
+  if (span.empty()) {
+    LOG_DEBUG("No span found, using default '%.*s'", (int)zerospan.length(), zerospan.data());
+    span = zerospan;
+  }
+
+  // span becomes new parent
+  ts::LocalBufferWriter<8192> bwriter;
+
+  bwriter.write(traceid);
+  bwriter.write(trace);
+  bwriter.write(sep);
+  bwriter.write(parentid);
+  bwriter.write(span);
+  bwriter.write(sep);
+  bwriter.write(spanid);
+  bwriter.print("{}", TSHttpTxnIdGet(txnp));
+
+  nexttrace.assign(bwriter.view());
+
+  return nexttrace;
+}
+
+std::string
+create_trace(TSHttpTxn const txnp)
+{
+  std::string header;
+
+  constexpr char new_parent{'0'};
+
+  TSUuid const uuid = TSUuidCreate();
+  if (nullptr != uuid) {
+    if (TS_SUCCESS == TSUuidInitialize(uuid, TS_UUID_V4)) {
+      char const *const uuidstr = TSUuidStringGet(uuid);
+      if (nullptr != uuidstr) {
+        ts::LocalBufferWriter<8192> bwriter;
+
+        bwriter.write(traceid);
+        bwriter.write(uuidstr);
+        bwriter.write(sep);
+        bwriter.write(parentid);
+        bwriter.write(new_parent);
+        bwriter.write(sep);
+        bwriter.write(spanid);
+        bwriter.print("{}", TSHttpTxnIdGet(txnp));
+
+        header.assign(bwriter.view());
+      } else {
+        LOG_ERROR("Error getting uuid string");
+      }
+    } else {
+      LOG_ERROR("Error initializing uuid");
+    }
+
+    TSUuidDestroy(uuid);
+  } else {
+    LOG_ERROR("Error calling TSUuidCreate");
+  }
+
+  return header;
 }
 
 /**
- * The TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE event callback.
- *
- * If there is a cache hit only schedule a TS_HTTP_SEND_RESPONSE_HDR_HOOK
- * continuation to send back the money trace header in the response to the
- * client.
- *
- * If there is a cache miss, a new money trace header is created and a
- * TS_HTTP_SEND_REQUES_HDR_HOOK continuation is scheduled to add the
- * new money trace header to the parent request.
+ * Creates header if necessary, sets given value.
  */
-static void
-mt_cache_lookup_check(TSCont contp, TSHttpTxn txnp, struct txndata *txn_data)
+bool
+set_header(TSMBuffer const bufp, TSMLoc const hdr_loc, std::string const &hdr, std::string const &val)
 {
-  MT generator;
-  int cache_result = 0;
-  char *new_mt_header;
+  bool isset = false;
 
-  if (TS_SUCCESS != TSHttpTxnCacheLookupStatusGet(txnp, &cache_result)) {
-    LOG_ERROR("Unable to get cache status.");
+  TSMLoc field_loc = TSMimeHdrFieldFind(bufp, hdr_loc, hdr.data(), hdr.length());
+
+  if (TS_NULL_MLOC == field_loc) {
+    // No existing header, so create one
+    if (TS_SUCCESS == TSMimeHdrFieldCreateNamed(bufp, hdr_loc, hdr.data(), hdr.length(), &field_loc)) {
+      if (TS_SUCCESS == TSMimeHdrFieldValueStringSet(bufp, hdr_loc, field_loc, -1, val.data(), val.length())) {
+        TSMimeHdrFieldAppend(bufp, hdr_loc, field_loc);
+        LOG_DEBUG("header/value added: '%s' '%s'", hdr.c_str(), val.c_str());
+        isset = true;
+      } else {
+        LOG_DEBUG("unable to set: '%s' to '%s'", hdr.c_str(), val.c_str());
+      }
+      TSHandleMLocRelease(bufp, hdr_loc, field_loc);
+    } else {
+      LOG_DEBUG("unable to create: '%s'", hdr.c_str());
+    }
   } else {
-    switch (cache_result) {
-    case TS_CACHE_LOOKUP_MISS:
-    case TS_CACHE_LOOKUP_SKIPPED:
-      new_mt_header = const_cast<char *>(generator.moneyTraceHdr(txn_data->client_request_mt_header));
-      if (new_mt_header != nullptr) {
-        LOG_DEBUG("cache miss, built a new money trace header: %s.", new_mt_header);
-        txn_data->new_span_mt_header = new_mt_header;
+    bool first = true;
+    while (field_loc) {
+      TSMLoc const tmp = TSMimeHdrFieldNextDup(bufp, hdr_loc, field_loc);
+      if (first) {
+        first = false;
+        if (TS_SUCCESS == TSMimeHdrFieldValueStringSet(bufp, hdr_loc, field_loc, -1, val.data(), val.length())) {
+          LOG_DEBUG("header/value set: '%s' '%s'", hdr.c_str(), val.c_str());
+          isset = true;
+        } else {
+          LOG_DEBUG("unable to set: '%s' to '%s'", hdr.c_str(), val.c_str());
+        }
       } else {
-        LOG_DEBUG("failed to build a new money trace header.");
+        TSMimeHdrFieldDestroy(bufp, hdr_loc, field_loc);
       }
-      TSHttpTxnHookAdd(txnp, TS_HTTP_SEND_REQUEST_HDR_HOOK, contp);
-
-      // fall through to the default as we always need to send the original
-      // money trace header received from the client back to the client in the
-      // response
-      // fallthrough
-
-    default:
-      TSHttpTxnHookAdd(txnp, TS_HTTP_SEND_RESPONSE_HDR_HOOK, contp);
-      break;
+      TSHandleMLocRelease(bufp, hdr_loc, field_loc);
+      field_loc = tmp;
     }
   }
+
+  return isset;
 }
 
 /**
- * remap entry point, called to check for the existence of a money trace
- * header.  If there is one, schedule the continuation to call back and
- * process on TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK and TS_HTTP_TXN_CLOSE_HOOK.
+ * The TS_EVENT_HTTP_POST_REMAP callback.
+ *
+ * If global_skip_header is set the global plugin
+ * will check for it here.
  */
-static void
-mt_check_request_header(TSHttpTxn txnp)
+void
+global_skip_check(TSCont const contp, TSHttpTxn const txnp, TxnData *const txn_data)
 {
-  int length               = 0;
-  struct txndata *txn_data = nullptr;
-  TSMBuffer bufp;
-  TSMLoc hdr_loc = nullptr, field_loc = nullptr;
-  TSCont contp;
+  Config const *const conf = txn_data->config;
+  if (conf->global_skip_header.empty()) {
+    LOG_ERROR("Called in error, no global skip header defined!");
+    return;
+  }
 
-  // check for a money trace header.  If there is one, schedule appropriate continuations.
+  // Check for a money trace header.  Route accordingly.
+  TSMBuffer bufp = nullptr;
+  TSMLoc hdr_loc = TS_NULL_MLOC;
   if (TS_SUCCESS == TSHttpTxnClientReqGet(txnp, &bufp, &hdr_loc)) {
-    field_loc = TSMimeHdrFieldFind(bufp, hdr_loc, MIME_FIELD_MONEY_TRACE, MIME_LEN_MONEY_TRACE);
+    TSMLoc const field_loc = TSMimeHdrFieldFind(bufp, hdr_loc, conf->global_skip_header.c_str(), conf->global_skip_header.length());
     if (TS_NULL_MLOC != field_loc) {
-      const char *hdr_value = TSMimeHdrFieldValueStringGet(bufp, hdr_loc, field_loc, 0, &length);
-      if (!hdr_value || length <= 0) {
-        LOG_DEBUG("ignoring, corrupt money trace header.");
-      } else {
-        if (nullptr == (contp = TSContCreate(transaction_handler, nullptr))) {
-          LOG_ERROR("failed to create the transaction handler continuation");
-        } else {
-          txn_data                                   = allocTransactionData();
-          txn_data->client_request_mt_header         = TSstrndup(hdr_value, length);
-          txn_data->client_request_mt_header[length] = '\0'; // workaround for bug in core.
-          LOG_DEBUG("found money trace header: %s, length: %d", txn_data->client_request_mt_header, length);
-          TSContDataSet(contp, txn_data);
-          TSHttpTxnHookAdd(txnp, TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK, contp);
-          TSHttpTxnHookAdd(txnp, TS_HTTP_TXN_CLOSE_HOOK, contp);
-        }
+      LOG_DEBUG("global_skip_header found, disabling for the rest of this transaction");
+      TSHandleMLocRelease(bufp, hdr_loc, field_loc);
+    } else { // schedule continuations
+      if (conf->create_if_none || !txn_data->client_trace.empty()) {
+        TSHttpTxnHookAdd(txnp, TS_HTTP_SEND_REQUEST_HDR_HOOK, contp);
       }
+      if (!txn_data->client_trace.empty()) {
+        TSHttpTxnHookAdd(txnp, TS_HTTP_SEND_RESPONSE_HDR_HOOK, contp);
+      }
+    }
+  }
+}
+
+/**
+ * The TS_EVENT_HTTP_SEND_REQUEST_HDR callback.
+ *
+ * When a parent request is made, this function adds the new
+ * money trace header to the parent request headers.
+ */
+void
+send_server_request(TSHttpTxn const txnp, TxnData *const txn_data)
+{
+  Config const *const conf = txn_data->config;
+  if (txn_data->this_trace.empty()) {
+    if (conf->passthru) {
+      txn_data->this_trace = txn_data->client_trace;
+    } else if (!txn_data->client_trace.empty()) {
+      txn_data->this_trace = next_trace(txn_data->client_trace, txnp);
+    } else if (conf->create_if_none) {
+      txn_data->this_trace = create_trace(txnp);
+    }
+  }
+
+  if (txn_data->this_trace.empty()) {
+    if (conf->create_if_none) {
+      LOG_DEBUG("Unable to deal with client trace '%s', creating new", txn_data->client_trace.c_str());
+      txn_data->this_trace = create_trace(txnp);
     } else {
-      LOG_DEBUG("no money trace header was found in the request.");
+      LOG_DEBUG("Unable to deal with client trace '%s', passing through!", txn_data->client_trace.c_str());
+      txn_data->this_trace = txn_data->client_trace;
+    }
+  }
+
+  TSMBuffer bufp = nullptr;
+  TSMLoc hdr_loc = TS_NULL_MLOC;
+  if (TS_SUCCESS == TSHttpTxnServerReqGet(txnp, &bufp, &hdr_loc)) {
+    if (!set_header(bufp, hdr_loc, txn_data->config->header, txn_data->this_trace)) {
+      LOG_ERROR("Unable to set the server request trace header '%s'", txn_data->this_trace.c_str());
     }
   } else {
-    LOG_DEBUG("failed to retrieve the client request.");
+    LOG_ERROR("Unable to get the txn server request");
   }
-  TSHandleMLocRelease(bufp, hdr_loc, field_loc);
 }
 
 /**
@@ -149,73 +380,164 @@ mt_check_request_header(TSHttpTxn txnp)
  * Adds the money trace header received in the client request to the
  * client response headers.
  */
-static void
-mt_send_client_response(TSHttpTxn txnp, struct txndata *txn_data)
+void
+send_client_response(TSHttpTxn const txnp, TxnData *const txn_data)
 {
-  TSMBuffer bufp;
-  TSMLoc hdr_loc = nullptr, field_loc = nullptr;
+  LOG_DEBUG("send_client_response");
 
-  if (txn_data->client_request_mt_header == nullptr) {
-    LOG_DEBUG("no client request header to return.");
+  if (txn_data->client_trace.empty()) {
+    LOG_DEBUG("no client trace data to return.");
     return;
   }
 
-  // send back the money trace header received in the request
-  // back in the response to the client.
-  if (TS_SUCCESS != TSHttpTxnClientRespGet(txnp, &bufp, &hdr_loc)) {
-    LOG_DEBUG("could not get the server response headers.");
-    return;
-  } else {
-    if (TS_SUCCESS ==
-        TSMimeHdrFieldCreateNamed(bufp, hdr_loc, MIME_FIELD_MONEY_TRACE, strlen(MIME_FIELD_MONEY_TRACE), &field_loc)) {
-      if (TS_SUCCESS == TSMimeHdrFieldValueStringSet(bufp, hdr_loc, field_loc, -1, txn_data->client_request_mt_header,
-                                                     strlen(txn_data->client_request_mt_header))) {
-        LOG_DEBUG("response header added: %s: %s", MIME_FIELD_MONEY_TRACE, txn_data->client_request_mt_header);
-        TSMimeHdrFieldAppend(bufp, hdr_loc, field_loc);
-      }
-    } else {
-      LOG_DEBUG("failed to create money trace response header.");
+  // send back the original money trace header received in the
+  // client request back in the response to the client.
+  TSMBuffer bufp = nullptr;
+  TSMLoc hdr_loc = TS_NULL_MLOC;
+  if (TS_SUCCESS == TSHttpTxnClientRespGet(txnp, &bufp, &hdr_loc)) {
+    if (!set_header(bufp, hdr_loc, txn_data->config->header, txn_data->client_trace)) {
+      LOG_ERROR("Unable to set the client response trace header.");
     }
+  } else {
+    LOG_DEBUG("Unable to get the txn client response");
   }
-  TSHandleMLocRelease(bufp, hdr_loc, field_loc);
-
-  return;
 }
 
 /**
- * The TS_EVENT_HTTP_SEND_REQUEST_HDR callback.
- *
- * When a parent request is made, this function adds the new
- * money trace header to the parent request headers.
+ * Transaction event handler.
  */
-static void
-mt_send_server_request(TSHttpTxn txnp, struct txndata *txn_data)
+int
+transaction_handler(TSCont const contp, TSEvent const event, void *const edata)
 {
-  TSMBuffer bufp;
-  TSMLoc hdr_loc = nullptr, field_loc = nullptr;
+  TSHttpTxn const txnp    = static_cast<TSHttpTxn>(edata);
+  TxnData *const txn_data = static_cast<TxnData *>(TSContDataGet(contp));
 
-  if (txn_data->new_span_mt_header == nullptr) {
-    LOG_DEBUG("there is no new mt request header to send.");
-    return;
+  switch (event) {
+  case TS_EVENT_HTTP_POST_REMAP:
+    LOG_DEBUG("global plugin checking for skip header");
+    global_skip_check(contp, txnp, txn_data);
+    break;
+  case TS_EVENT_HTTP_SEND_REQUEST_HDR:
+    LOG_DEBUG("updating send request headers.");
+    send_server_request(txnp, txn_data);
+    break;
+  case TS_EVENT_HTTP_SEND_RESPONSE_HDR:
+    LOG_DEBUG("updating send response headers.");
+    send_client_response(txnp, txn_data);
+    break;
+  case TS_EVENT_HTTP_TXN_CLOSE:
+    LOG_DEBUG("handling transaction close.");
+    delete txn_data;
+    TSContDestroy(contp);
+    break;
+  default:
+    TSAssert(!"Unexpected event");
+    break;
   }
+  TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE);
 
-  if (TS_SUCCESS == TSHttpTxnServerReqGet(txnp, &bufp, &hdr_loc)) {
-    field_loc = TSMimeHdrFieldFind(bufp, hdr_loc, MIME_FIELD_MONEY_TRACE, MIME_LEN_MONEY_TRACE);
+  return TS_SUCCESS;
+}
+
+/**
+ * Check for the existence of a money trace header.
+ * If there is one, schedule the continuation to call back and
+ * process on send request or send response.
+ * Global plugin may need to schedule a hook to check for skip header.
+ */
+void
+check_request_header(TSHttpTxn const txnp, Config const *const conf, PluginType const ptype)
+{
+  TxnData *txn_data = nullptr;
+
+  // Check for a money trace header.  Route accordingly.
+  TSMBuffer bufp = nullptr;
+  TSMLoc hdr_loc = TS_NULL_MLOC;
+  if (TS_SUCCESS == TSHttpTxnClientReqGet(txnp, &bufp, &hdr_loc)) {
+    TSMLoc const field_loc = TSMimeHdrFieldFind(bufp, hdr_loc, conf->header.c_str(), conf->header.length());
     if (TS_NULL_MLOC != field_loc) {
-      if (TS_SUCCESS == TSMimeHdrFieldValueStringSet(bufp, hdr_loc, field_loc, -1, txn_data->new_span_mt_header,
-                                                     strlen(txn_data->new_span_mt_header))) {
-        LOG_DEBUG("server request header updated: %s: %s", MIME_FIELD_MONEY_TRACE, txn_data->new_span_mt_header);
+      int length            = 0;
+      const char *hdr_value = TSMimeHdrFieldValueStringGet(bufp, hdr_loc, field_loc, 0, &length);
+      if (!hdr_value || length <= 0) {
+        LOG_DEBUG("ignoring, corrupt trace header.");
+      } else {
+        txn_data         = new TxnData;
+        txn_data->config = conf;
+        txn_data->client_trace.assign(hdr_value, length);
+        LOG_DEBUG("found monetrace header: '%.*s', length: %d", length, hdr_value, length);
+      }
+      TSHandleMLocRelease(bufp, hdr_loc, field_loc);
+    } else if (!conf->passthru && conf->create_if_none) {
+      txn_data             = new TxnData;
+      txn_data->config     = conf;
+      txn_data->this_trace = create_trace(txnp);
+      LOG_DEBUG("created trace header: '%s'", txn_data->this_trace.c_str());
+    } else {
+      LOG_DEBUG("no trace header handling for this request.");
+    }
+
+    // Check for pregen_header
+    if (nullptr != txn_data && !conf->pregen_header.empty()) {
+      if (txn_data->this_trace.empty()) {
+        txn_data->this_trace = next_trace(txn_data->client_trace, txnp);
+
+        if (txn_data->this_trace.empty()) {
+          if (conf->create_if_none) {
+            LOG_DEBUG("Unable to deal with client trace '%s', creating new", txn_data->client_trace.c_str());
+            txn_data->this_trace = create_trace(txnp);
+          } else {
+            LOG_DEBUG("Unable to deal with client trace '%s', passing through!", txn_data->client_trace.c_str());
+            txn_data->this_trace = txn_data->client_trace;
+          }
+        }
+      }
+      if (!txn_data->this_trace.empty()) {
+        if (!set_header(bufp, hdr_loc, conf->pregen_header, txn_data->this_trace)) {
+          LOG_ERROR("Unable to set the client request pregen trace header.");
+        }
+      }
+    }
+  } else {
+    LOG_DEBUG("unable to get the request request");
+  }
+
+  // Schedule appropriate continuations
+  if (nullptr != txn_data) {
+    TSCont const contp = TSContCreate(transaction_handler, nullptr);
+    if (nullptr != contp) {
+      TSContDataSet(contp, txn_data);
+      TSHttpTxnHookAdd(txnp, TS_HTTP_TXN_CLOSE_HOOK, contp);
+
+      // global plugin may need to check for skip header
+      if (GLOBAL == ptype && !conf->global_skip_header.empty()) {
+        TSHttpTxnHookAdd(txnp, TS_HTTP_POST_REMAP_HOOK, contp);
+      } else {
+        if (conf->create_if_none || !txn_data->client_trace.empty()) {
+          TSHttpTxnHookAdd(txnp, TS_HTTP_SEND_REQUEST_HDR_HOOK, contp);
+        }
+        if (!txn_data->client_trace.empty()) {
+          TSHttpTxnHookAdd(txnp, TS_HTTP_SEND_RESPONSE_HDR_HOOK, contp);
+        }
       }
     } else {
-      LOG_DEBUG("unable to retrieve the money trace header location from the server request headers.");
-      return;
+      LOG_ERROR("failed to create the transaction handler continuation");
+      delete txn_data;
     }
   }
-  TSHandleMLocRelease(bufp, hdr_loc, field_loc);
+}
 
-  return;
+int
+global_request_header_hook(TSCont const contp, TSEvent const event, void *const edata)
+{
+  TSHttpTxn const txnp     = static_cast<TSHttpTxn>(edata);
+  Config const *const conf = static_cast<Config *>(TSContDataGet(contp));
+  check_request_header(txnp, conf, GLOBAL);
+  TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE);
+  return 0;
 }
 
+} // namespace
+
 /**
  * Remap initialization.
  */
@@ -233,7 +555,7 @@ TSRemapInit(TSRemapInterface *api_info, char *errbuf, int errbuf_size)
     return TS_ERROR;
   }
 
-  LOG_DEBUG("cache_range_requests remap is successfully initialized.");
+  LOG_DEBUG("money_trace remap is successfully initialized.");
 
   return TS_SUCCESS;
 }
@@ -244,6 +566,11 @@ TSRemapInit(TSRemapInterface *api_info, char *errbuf, int errbuf_size)
 TSReturnCode
 TSRemapNewInstance(int argc, char *argv[], void **ih, char * /*errbuf */, int /* errbuf_size */)
 {
+  // second arg poses as the program name
+  --argc;
+  ++argv;
+  Config *const conf = config_from_args(argc, const_cast<char const **>(argv), REMAP);
+  *ih                = static_cast<void *>(conf);
   return TS_SUCCESS;
 }
 
@@ -253,7 +580,8 @@ TSRemapNewInstance(int argc, char *argv[], void **ih, char * /*errbuf */, int /*
 void
 TSRemapDeleteInstance(void *ih)
 {
-  LOG_DEBUG("no op");
+  Config *const conf = static_cast<Config *>(ih);
+  delete conf;
 }
 
 /**
@@ -262,97 +590,34 @@ TSRemapDeleteInstance(void *ih)
 TSRemapStatus
 TSRemapDoRemap(void *ih, TSHttpTxn txnp, TSRemapRequestInfo * /* rri */)
 {
-  mt_check_request_header(txnp);
+  Config const *const conf = static_cast<Config *>(ih);
+  check_request_header(txnp, conf, REMAP);
   return TSREMAP_NO_REMAP;
 }
 
 /**
- * Transaction event handler.
+ * global plugin initialization
  */
-static int
-transaction_handler(TSCont contp, TSEvent event, void *edata)
-{
-  TSHttpTxn txnp           = static_cast<TSHttpTxn>(edata);
-  struct txndata *txn_data = static_cast<struct txndata *>(TSContDataGet(contp));
-
-  switch (event) {
-  case TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE:
-    LOG_DEBUG("transaction cache lookup complete.");
-    mt_cache_lookup_check(contp, txnp, txn_data);
-    break;
-  case TS_EVENT_HTTP_SEND_REQUEST_HDR:
-    LOG_DEBUG("updating send request headers.");
-    mt_send_server_request(txnp, txn_data);
-    break;
-  case TS_EVENT_HTTP_SEND_RESPONSE_HDR:
-    LOG_DEBUG("updating send response headers.");
-    mt_send_client_response(txnp, txn_data);
-    break;
-  case TS_EVENT_HTTP_TXN_CLOSE:
-    LOG_DEBUG("handling transaction close.");
-    freeTransactionData(txn_data);
-    TSContDestroy(contp);
-    break;
-  default:
-    TSAssert(!"Unexpected event");
-    break;
-  }
-  TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE);
-
-  return TS_SUCCESS;
-}
-
-const char *
-MT::moneyTraceHdr(const char *mt_request_hdr)
+void
+TSPluginInit(int argc, char const *argv[])
 {
-  char copy[8192] = {'\0'};
-  char *toks[3], *p = nullptr, *saveptr = nullptr;
-  std::ostringstream temp_str;
-  std::string mt_header_str;
-  int numtoks = 0;
-
-  if (mt_request_hdr == nullptr) {
-    LOG_DEBUG("an empty header was passed in.");
-    return nullptr;
-  } else {
-    strncpy(copy, mt_request_hdr, 8191);
-  }
+  LOG_DEBUG("Starting global plugin init");
 
-  // parse the money header.
-  p = strtok_r(copy, ";", &saveptr);
-  if (p != nullptr) {
-    toks[numtoks++] = p;
-    // copy the traceid
-  } else {
-    LOG_DEBUG("failed to parse the money_trace_header: %s", mt_request_hdr);
-    return nullptr;
-  }
-  do {
-    p = strtok_r(nullptr, ";", &saveptr);
-    if (p != nullptr) {
-      toks[numtoks++] = p;
-    }
-  } while (p != nullptr && numtoks < 3);
+  TSPluginRegistrationInfo info;
+  info.plugin_name   = PLUGIN_NAME;
+  info.vendor_name   = "Apache Software Foundation";
+  info.support_email = "dev@trafficserver.apache.org";
 
-  if (numtoks != 3 || toks[0] == nullptr || toks[1] == nullptr || toks[2] == nullptr) {
-    LOG_DEBUG("failed to parse the money_trace_header: %s", mt_request_hdr);
-    return nullptr;
+  if (TS_SUCCESS != TSPluginRegister(&info)) {
+    LOG_ERROR("Plugin registration failed");
+    return;
   }
 
-  if (strncmp(toks[0], "trace-id", strlen("trace-id")) == 0 && strncmp(toks[2], "span-id", strlen("span-id")) == 0 &&
-      (p = strchr(toks[2], '=')) != nullptr) {
-    p++;
-    if (strncmp("0x", p, 2) == 0) {
-      temp_str << toks[0] << ";parent-id=" << p << ";span-id=0x" << std::hex << spanId() << std::ends;
-    } else {
-      temp_str << toks[0] << ";parent-id=" << p << ";span-id=" << spanId() << std::ends;
-    }
-  } else {
-    LOG_DEBUG("invalid money_trace_header: %s", mt_request_hdr);
-    return nullptr;
-  }
+  Config const *const conf = config_from_args(argc, argv, GLOBAL);
 
-  mt_header_str = temp_str.str();
+  TSCont const contp = TSContCreate(global_request_header_hook, nullptr);
+  TSContDataSet(contp, (void *)conf);
 
-  return TSstrndup(mt_header_str.c_str(), mt_header_str.length());
+  // This fires before any remap hooks.
+  TSHttpHookAdd(TS_HTTP_READ_REQUEST_HDR_HOOK, contp);
 }
diff --git a/plugins/experimental/money_trace/money_trace.h b/plugins/experimental/money_trace/money_trace.h
index 88af66b64..f98e7eb25 100644
--- a/plugins/experimental/money_trace/money_trace.h
+++ b/plugins/experimental/money_trace/money_trace.h
@@ -19,8 +19,6 @@
 
 #pragma once
 
-#include <random>
-#include <ctime>
 #include "ts/ts.h"
 #include "ts/remap.h"
 
@@ -35,32 +33,3 @@
   do {                                                                          \
     TSError("[%s:%d] %s(): " fmt, __FILE__, __LINE__, __func__, ##__VA_ARGS__); \
   } while (0)
-
-#define MIME_FIELD_MONEY_TRACE "X-MoneyTrace"
-#define MIME_LEN_MONEY_TRACE 12
-
-struct MT {
-  std::minstd_rand0 generator;
-
-  MT() { generator.seed(time(nullptr)); }
-  long
-  spanId()
-  {
-    long v = generator();
-    return (v * v);
-  }
-  const char *moneyTraceHdr(const char *mt_request_hdr);
-};
-
-struct txndata {
-  char *client_request_mt_header;
-  char *new_span_mt_header;
-};
-
-static struct txndata *allocTransactionData();
-static void freeTransactionData(struct txndata *txn_data);
-static void mt_cache_lookup_check(TSCont contp, TSHttpTxn txnp, struct txndata *txn_data);
-static void mt_check_request_header(TSHttpTxn txnp);
-static void mt_send_client_response(TSHttpTxn txnp, struct txndata *txn_data);
-static void mt_send_server_request(TSHttpTxn txnp, struct txndata *txn_data);
-static int transaction_handler(TSCont contp, TSEvent event, void *edata);
diff --git a/tests/gold_tests/pluginTest/money_trace/gold/global-log.gold b/tests/gold_tests/pluginTest/money_trace/gold/global-log.gold
new file mode 100644
index 000000000..486db70dd
--- /dev/null
+++ b/tests/gold_tests/pluginTest/money_trace/gold/global-log.gold
@@ -0,0 +1,3 @@
+cqh: trace-id=foo;parent-id=bar;span-id=baz trace-id=foo;parent-id=baz;span-id=0 pqh: trace-id=foo;parent-id=baz;span-id=0 psh: trace-id=foo;parent-id=bar;span-id=baz
+cqh: - ``;parent-id=0;span-id=1 pqh: - psh: -
+cqh: - trace-id=``;parent-id=0;span-id=2 pqh: trace-id=``;parent-id=0;span-id=2 psh: -
diff --git a/tests/gold_tests/pluginTest/money_trace/gold/remap-log.gold b/tests/gold_tests/pluginTest/money_trace/gold/remap-log.gold
new file mode 100644
index 000000000..c8c98b53c
--- /dev/null
+++ b/tests/gold_tests/pluginTest/money_trace/gold/remap-log.gold
@@ -0,0 +1,23 @@
+cqh: - - - pqh: - - psh: - -
+cqh: - - - pqh: - - psh: - -
+cqh: trace-id=basic;parent-id=foo;span-id=bar - - pqh: trace-id=basic;parent-id=bar;span-id=2 - psh: trace-id=basic;parent-id=foo;span-id=bar -
+cqh: - trace-id=header;parent-id=foo;span-id=bar - pqh: - trace-id=header;parent-id=bar;span-id=3 psh: - trace-id=header;parent-id=foo;span-id=bar
+cqh: - - - pqh: - - psh: - -
+cqh: trace-id=pregen;parent-id=foo;span-id=bar - trace-id=pregen;parent-id=bar;span-id=5 pqh: trace-id=pregen;parent-id=bar;span-id=5 - psh: trace-id=pregen;parent-id=foo;span-id=bar -
+cqh: - trace-id=pgh;parent-id=foo;span-id=bar trace-id=pgh;parent-id=bar;span-id=6 pqh: - trace-id=pgh;parent-id=bar;span-id=6 psh: - trace-id=pgh;parent-id=foo;span-id=bar
+cqh: - - - pqh: trace-id=``;parent-id=0;span-id=7 - psh: - -
+cqh: - - - pqh: - trace-id=``;parent-id=0;span-id=8 psh: - -
+cqh: - - trace-id=``;parent-id=0;span-id=9 pqh: trace-id=``;parent-id=0;span-id=9 - psh: - -
+cqh: trace-id=passthru;parent-id=foo;span-id=bar - - pqh: trace-id=passthru;parent-id=foo;span-id=bar - psh: trace-id=passthru;parent-id=foo;span-id=bar -
+cqh: trace-id=spaces; parent-id=foo; span-id=bar - trace-id=spaces;parent-id=bar;span-id=11 pqh: trace-id=spaces;parent-id=bar;span-id=11 - psh: trace-id=spaces; parent-id=foo; span-id=bar -
+cqh: trace-id=noparent;span-id=8215111;span-name=this is some description - trace-id=noparent;parent-id=8215111;span-id=12 pqh: trace-id=noparent;parent-id=8215111;span-id=12 - psh: trace-id=noparent;span-id=8215111;span-name=this is some description -
+cqh: trace-id=badspan;span-id= - trace-id=badspan;parent-id=0;span-id=13 pqh: trace-id=badspan;parent-id=0;span-id=13 - psh: trace-id=badspan;span-id= -
+cqh: trace-id=traceonly - trace-id=traceonly;parent-id=0;span-id=14 pqh: trace-id=traceonly;parent-id=0;span-id=14 - psh: trace-id=traceonly -
+cqh: trace-id=traceonlysemi; - trace-id=traceonlysemi;parent-id=0;span-id=15 pqh: trace-id=traceonlysemi;parent-id=0;span-id=15 - psh: trace-id=traceonlysemi; -
+cqh: not a trace header - not a trace header pqh: not a trace header - psh: not a trace header -
+cqh: trace-id=spaces; parent-id=foo; span-id=bar - trace-id=spaces;parent-id=bar;span-id=17 pqh: trace-id=spaces;parent-id=bar;span-id=17 - psh: trace-id=spaces; parent-id=foo; span-id=bar -
+cqh: trace-id=noparent;span-id=8215111;span-name=this is some description - trace-id=noparent;parent-id=8215111;span-id=18 pqh: trace-id=noparent;parent-id=8215111;span-id=18 - psh: trace-id=noparent;span-id=8215111;span-name=this is some description -
+cqh: trace-id=badspan;span-id= - trace-id=badspan;parent-id=0;span-id=19 pqh: trace-id=badspan;parent-id=0;span-id=19 - psh: trace-id=badspan;span-id= -
+cqh: trace-id=traceonly - trace-id=traceonly;parent-id=0;span-id=20 pqh: trace-id=traceonly;parent-id=0;span-id=20 - psh: trace-id=traceonly -
+cqh: trace-id=traceonlysemi; - trace-id=traceonlysemi;parent-id=0;span-id=21 pqh: trace-id=traceonlysemi;parent-id=0;span-id=21 - psh: trace-id=traceonlysemi; -
+cqh: not a trace header - trace-id=``;parent-id=0;span-id=22 pqh: trace-id=``;parent-id=0;span-id=22 - psh: not a trace header -
diff --git a/tests/gold_tests/pluginTest/money_trace/money_trace.test.py b/tests/gold_tests/pluginTest/money_trace/money_trace.test.py
new file mode 100644
index 000000000..5b35c1421
--- /dev/null
+++ b/tests/gold_tests/pluginTest/money_trace/money_trace.test.py
@@ -0,0 +1,225 @@
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+import os
+Test.Summary = '''
+Test money_trace remap
+'''
+
+# Test description:
+
+Test.SkipUnless(
+    Condition.PluginExists('money_trace.so'),
+)
+Test.ContinueOnFail = False
+Test.testName = "money_trace remap"
+
+# Define ATS and configure
+ts = Test.MakeATSProcess("ts", command="traffic_server", enable_cache=False)
+
+# configure origin server
+server = Test.MakeOriginServer("server")
+
+req_chk = {"headers":
+           "GET / HTTP/1.1\r\n" + "Host: origin\r\n" + "\r\n",
+           "timestamp": "1469733493.993",
+           "body": ""
+           }
+res_chk = {"headers":
+           "HTTP/1.1 200 OK\r\n" + "Connection: close\r\n" + "\r\n",
+           "timestamp": "1469733493.993",
+           "body": ""
+           }
+server.addResponse("sessionlog.json", req_chk, res_chk)
+
+req_hdr = {"headers":
+           "GET /path HTTP/1.1\r\n" + "Host: www.example.com\r\n" + "\r\n",
+           "timestamp": "1469733493.993",
+           "body": ""
+           }
+res_hdr = {"headers":
+           "HTTP/1.1 200 OK\r\n" + "Connection: close\r\n" + "\r\n",
+           "timestamp": "1469733493.993",
+           "body": ""
+           }
+server.addResponse("sessionlog.json", req_hdr, res_hdr)
+
+ts.Disk.remap_config.AddLines([
+    f"map http://none/ http://127.0.0.1:{server.Variables.Port}",
+    f"map http://basic/ http://127.0.0.1:{server.Variables.Port} @plugin=money_trace.so",
+    f"map http://header/ http://127.0.0.1:{server.Variables.Port} @plugin=money_trace.so @pparam=--header=mt",
+    f"map http://pregen/ http://127.0.0.1:{server.Variables.Port} @plugin=money_trace.so @pparam=--pregen-header=@pregen",
+    f"map http://pgh/ http://127.0.0.1:{server.Variables.Port} @plugin=money_trace.so @pparam=--header=mt @pparam=--pregen-header=@pregen",
+    f"map http://create/ http://127.0.0.1:{server.Variables.Port} @plugin=money_trace.so @pparam=--create-if-none=true",
+    f"map http://cheader/ http://127.0.0.1:{server.Variables.Port} @plugin=money_trace.so @pparam=--create-if-none=true @pparam=--header=mt",
+    f"map http://cpregen/ http://127.0.0.1:{server.Variables.Port} @plugin=money_trace.so @pparam=--create-if-none=true @pparam=--pregen-header=@pregen",
+    f"map http://passthru/ http://127.0.0.1:{server.Variables.Port} @plugin=money_trace.so @pparam=--passthru=true",
+])
+
+# minimal configuration
+ts.Disk.records_config.update({
+    'proxy.config.diags.debug.enabled': 0,
+    'proxy.config.diags.debug.tags': 'money_trace',
+})
+
+ts.Disk.logging_yaml.AddLines(
+    '''
+logging:
+  formats:
+    - name: custom
+      format: 'cqh: %<{X-MoneyTrace}cqh> %<{mt}cqh> %<{@pregen}cqh> pqh: %<{X-MoneyTrace}pqh> %<{mt}pqh> psh: %<{X-MoneyTrace}psh> %<{mt}psh>'
+  logs:
+    - filename: remap
+      format: custom
+'''.split("\n")
+)
+
+Test.Disk.File(os.path.join(ts.Variables.LOGDIR, 'remap.log'),
+               exists=True, content='gold/remap-log.gold')
+
+curl_and_args = f"curl -s -D /dev/stdout -o /dev/stderr -x 127.0.0.1:{ts.Variables.port}"
+
+# 0 Test
+tr = Test.AddTestRun("no plugin test")
+ps = tr.Processes.Default
+ps.StartBefore(server)
+ps.StartBefore(Test.Processes.ts)
+ps.Command = curl_and_args + " http://none/path"
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 1 Test
+tr = Test.AddTestRun("basic config, no money trace client header")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + " http://basic/path"
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+
+def maketrace(name):
+    return f'trace-id={name};parent-id=foo;span-id=bar'
+
+
+# 2 Test
+tr = Test.AddTestRun("basic config, money trace client header")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://basic/path -H "X-MoneyTrace: ' + maketrace("basic") + '"'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 3 Test
+tr = Test.AddTestRun("header config, mt client header")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://header/path -H "mt: ' + maketrace("header") + '"'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 4 Test
+tr = Test.AddTestRun("pregen config, but no header passed in")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://pregen/path'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 5 Test
+tr = Test.AddTestRun("pregen config, money trace client header")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://pregen/path -H "X-MoneyTrace: ' + maketrace("pregen") + '"'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 6 Test
+tr = Test.AddTestRun("pregen config, mt client header")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://pgh/path -H "mt: ' + maketrace("pgh") + '"'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 7 Test
+tr = Test.AddTestRun("create config, money trace client header")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://create/path'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 8 Test
+tr = Test.AddTestRun("create config, mt client header")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://cheader/path'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 9 Test
+tr = Test.AddTestRun("create config, pregen client header")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://cpregen/path'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 10 Test
+tr = Test.AddTestRun("passthru mode")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://passthru/path -H "X-MoneyTrace: ' + maketrace("passthru") + '"'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# Parsing robustness
+
+trace_strings = [
+    "trace-id=spaces; parent-id=foo; span-id=bar",
+    "trace-id=noparent;span-id=8215111;span-name=this is some description",
+    "trace-id=badspan;span-id=",
+    "trace-id=traceonly",
+    "trace-id=traceonlysemi;",
+    "not a trace header",
+]
+
+# 11 Test
+for trace in trace_strings:
+    tr = Test.AddTestRun(trace)
+    ps = tr.Processes.Default
+    ps.Command = curl_and_args + ' http://pregen/path -H "X-MoneyTrace: ' + trace + '"'
+    ps.ReturnCode = 0
+    tr.StillRunningAfter = ts
+    tr.StillRunningAfter = server
+
+# 11 Test
+for trace in trace_strings:
+    tr = Test.AddTestRun(trace)
+    ps = tr.Processes.Default
+    ps.Command = curl_and_args + ' http://cpregen/path -H "X-MoneyTrace: ' + trace + '"'
+    ps.ReturnCode = 0
+    tr.StillRunningAfter = ts
+    tr.StillRunningAfter = server
+
+# Wait for log file to appear, then wait one extra second to make sure TS is done writing it.
+# 11 Test
+tr = Test.AddTestRun()
+ps = tr.Processes.Default
+ps.Command = (
+    os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' +
+    os.path.join(ts.Variables.LOGDIR, 'remap.log')
+)
diff --git a/tests/gold_tests/pluginTest/money_trace/money_trace_global.test.py b/tests/gold_tests/pluginTest/money_trace/money_trace_global.test.py
new file mode 100644
index 000000000..d7092d85e
--- /dev/null
+++ b/tests/gold_tests/pluginTest/money_trace/money_trace_global.test.py
@@ -0,0 +1,126 @@
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+import os
+Test.Summary = '''
+Test money_trace global
+'''
+
+# Test description:
+
+Test.SkipUnless(
+    #    Condition.PluginExists('xdebug.so'),
+    Condition.PluginExists('money_trace.so'),
+)
+Test.ContinueOnFail = False
+Test.testName = "money_trace global"
+
+# Define ATS and configure
+ts = Test.MakeATSProcess("ts", command="traffic_server", enable_cache=False)
+
+# configure origin server
+server = Test.MakeOriginServer("server")
+
+req_chk = {"headers":
+           "GET / HTTP/1.1\r\n" + "Host: origin\r\n" + "\r\n",
+           "timestamp": "1469733493.993",
+           "body": ""
+           }
+res_chk = {"headers":
+           "HTTP/1.1 200 OK\r\n" + "Connection: close\r\n" + "\r\n",
+           "timestamp": "1469733493.993",
+           "body": ""
+           }
+server.addResponse("sessionlog.json", req_chk, res_chk)
+
+req_hdr = {"headers":
+           "GET /path HTTP/1.1\r\n" + "Host: www.example.com\r\n" + "\r\n",
+           "timestamp": "1469733493.993",
+           "body": ""
+           }
+res_hdr = {"headers":
+           "HTTP/1.1 200 OK\r\n" + "Connection: close\r\n" + "\r\n",
+           "timestamp": "1469733493.993",
+           "body": ""
+           }
+server.addResponse("sessionlog.json", req_hdr, res_hdr)
+
+ts.Disk.remap_config.AddLines([
+    f"map http://ats http://127.0.0.1:{server.Variables.Port}",
+])
+
+ts.Disk.plugin_config.AddLine(
+    "money_trace.so --pregen-header=@pregen --header=MoneyTrace --create-if-none=true --global-skip-header=Skip-Global-MoneyTrace")
+
+# minimal configuration
+ts.Disk.records_config.update({
+    'proxy.config.diags.debug.enabled': 0,
+    'proxy.config.diags.debug.tags': 'money_trace',
+})
+
+ts.Disk.logging_yaml.AddLines(
+    '''
+logging:
+  formats:
+    - name: custom
+      format: 'cqh: %<{MoneyTrace}cqh> %<{@pregen}cqh> pqh: %<{MoneyTrace}pqh> psh: %<{MoneyTrace}psh>'
+  logs:
+    - filename: global
+      format: custom
+'''.split("\n")
+)
+
+Test.Disk.File(os.path.join(ts.Variables.LOGDIR, 'global.log'),
+               exists=True, content='gold/global-log.gold')
+
+curl_and_args = f"curl -s -D /dev/stdout -o /dev/stderr -x 127.0.0.1:{ts.Variables.port}"  # -H 'X-Debug: Probe' "
+
+clientvalue = "trace-id=foo;parent-id=bar;span-id=baz"
+
+# 0 Test
+tr = Test.AddTestRun("normal header test")
+ps = tr.Processes.Default
+ps.StartBefore(server, ready=When.PortOpen(server.Variables.Port))
+ps.StartBefore(Test.Processes.ts)
+ps.Command = curl_and_args + ' http://ats/path -H "MoneyTrace: ' + clientvalue + '"'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 1 Test
+tr = Test.AddTestRun("skip plugin test - pregen will still be set")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://ats/path -H "Skip-Global-MoneyTrace: true"'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# 2 Test
+tr = Test.AddTestRun("create header test")
+ps = tr.Processes.Default
+ps.Command = curl_and_args + ' http://ats/path'
+ps.ReturnCode = 0
+tr.StillRunningAfter = ts
+tr.StillRunningAfter = server
+
+# Wait for log file to appear, then wait one extra second to make sure TS is done writing it.
+tr = Test.AddTestRun()
+ps = tr.Processes.Default
+ps.Command = (
+    os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' +
+    os.path.join(ts.Variables.LOGDIR, 'global.log')
+)
+#ps.ReturnCode = 0