You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficserver.apache.org by cm...@apache.org on 2026/09/09 15:14:06 UTC

(trafficserver) branch master updated: Metrics: close the id lookup race and bounds gaps left by the lock revert (#13583)

This is an automated email from the ASF dual-hosted git repository.

cmcfarlen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/master by this push:
     new c7af2e39b5 Metrics: close the id lookup race and bounds gaps left by the lock revert (#13583)
c7af2e39b5 is described below

commit c7af2e39b5c2a9fb3a26c2dc4bbb38d26d60c7e2
Author: Chris McFarlen <ch...@mcfarlen.us>
AuthorDate: Wed Sep 9 10:13:59 2026 -0500

    Metrics: close the id lookup race and bounds gaps left by the lock revert (#13583)
    
    * Metrics: one gate for id validation, and fix the off-by-one
    
    valid(), lookup(IdType), name() and rename() each carried their own copy of
    the same range test, and the copies had drifted. valid() rejected an offset
    past MAX_SIZE; the other three did not. Since _splitID passes the low 16
    bits of an id through unmasked and the offset check only applied when the id
    named the current blob, an id such as 0x0000FFFF indexed well past the end
    of a blob's 1024 entry arrays once a second blob existed. Ids reaching these
    accessors come from plugins through the TSStat* API, so they are untrusted.
    
    All four now go through Storage::_is_allocated(), which rejects a negative
    id, an offset no _makeId could have produced, an unallocated blob, and a
    slot at or past the allocation point. That last comparison also fixes an
    off-by-one: create() returns the id and then advances, so _cur_off is the
    next free slot, and the old <= / > tests accepted it. An increment there
    landed on the slot create() would hand out next, and since create() writes
    only the name and never the value, the next plugin to call TSStatCreate()
    received a metric already carrying someone else's count.
    
    Nothing depended on the loose bound: end() builds an id at the allocation
    point that is compared but never dereferenced, iterator::next() keeps the
    offset in range, and find() returns end() on a miss.
    
    * Metrics: publish the allocation point with release/acquire
    
    The lock removal in #13567 left the reader path reading _cur_blob, _cur_off
    and _blobs while a concurrent create() advances them, which is the data race
    #13310 took the mutex to close. Close it without the mutex instead.
    
    Making each counter atomic does not make the pair update atomically, and it
    does not need to. _cur_blob and _cur_off are publication points: each is
    written last, with a release store, after whatever it makes visible -- the
    blob pointer and the reset offset for _cur_blob, the slot's name for
    _cur_off. A reader acquires _cur_blob first, so observing a value for it also
    observes everything addBlob() wrote before releasing it. The torn pair a
    reader could otherwise see, a new blob index with the previous blob's stale
    offset, is unreachable rather than merely unlikely, so neither a packed word
    nor per-blob counters are needed.
    
    _blobs stays non-atomic. It is only read at an index no greater than
    _cur_blob, and that write is sequenced before the release store the reader
    acquired, so there is no race to close.
    
    Writers all hold the mutex and load relaxed. What remains is that a reader
    can observe an older _cur_blob with an already reset _cur_off and reject an
    id naming the previous blob, which drops an increment rather than
    misattributing one.
    
    Verified with a TSAN harness running eight readers validating and resolving
    ids across the whole space while a writer creates 2600 metrics across
    several blob boundaries: three reported races before this change, none
    after.
    
    * Metrics: cover concurrent id lookup, and make _extractType total
    
    Add a test that resolves ids from several threads while another registers
    metrics across a few blob boundaries. Nothing single threaded exercises the
    publication order the previous commit relies on; under the tsan preset,
    making either allocation counter non-atomic again reports a data race here.
    The test cannot catch a downgrade of the release/acquire pairs to relaxed --
    atomics are race free at any ordering -- and says so, so the memory orders
    are not mistaken for tested.
    
    _extractType shifted a signed IdType, so _extractType(NOT_FOUND) sign
    extended to -4, a MetricType outside its enumeration, returned by
    Metrics::type(). Shifting unsigned is not enough on its own: the sign bit
    sits above the type field, so NOT_FOUND still yields 4. Mask to the single
    bit _makeId writes, which makes the function total for any input.
    
    * Add a ts::Metrics micro benchmark
    
    Nothing in tree measured the metric read paths, which is why a global mutex
    on the hottest one went unnoticed until it showed up in a production
    profile. Four cases, scaled by thread count:
    
      increment(id)   what TSStatIntIncrement does, the path that regressed
      increment(ptr)  what core and cripts do, the floor
      lookup(id)      the lock free id resolution alone
      lookup(name)    the same resolution through the mutex guarded name map
    
    lookup(name) is deliberately included as a positive control. It still takes
    the lock, so it must degrade with thread count; if it ever stops doing so,
    the harness is not loading the machine and the other three numbers mean
    nothing.
    
    Built only with ENABLE_BENCHMARKS, as with the rest of tools/benchmark.
    
    * Metrics: trim comments to the invariants
    
    State what holds rather than how it came to hold. Drops the explanations of
    which write order a comparison compensates for, what a reader would have
    seen otherwise, and what each benchmark case is meant to prove.
    
    Also shortens the createSpan boundary test's preamble, which describes the
    bug it covers at more length than the assertion needs.
    
    * Metrics: probe real ids in the concurrent lookup test
    
    The reader swept ids as consecutive integers, but an id packs the blob index
    above the offset, so 0..N only ever named blob 0 and everything from
    MAX_SIZE up decoded to an offset that validation rejects. Earlier cases in
    this file leave blob 0 full, so the reader was walking settled slots while
    the writer worked in a blob it never named.
    
    Take ids from what the writer has registered instead, and assert the ids
    span more than one blob so a future change cannot quietly confine the sweep
    again. Also assert the readers resolved something, since every id being
    skipped would otherwise pass.
    
    * Metrics: make _is_allocated private, tidy createSpan's index handling
    
    _is_allocated is only called by Storage's own accessors, so it does not
    belong in the public section; valid() remains the public gate.
    
    createSpan loaded _cur_off twice and _cur_blob once for its two guards, then
    re-read both unconditionally in case addBlob() had moved them. Load the pair
    once and refresh it only in the branch that grows a blob, which drops two
    atomic loads from the common path. Re-reading rather than adjusting the
    locals by hand keeps the caller from restating what addBlob() sets.
    
    * Publish the next free slot as one packed value
    
    _cur_blob and _cur_off were two atomics, and readers need the pair to be
    coherent. addBlob() reset the offset then bumped the blob, so a reader
    between the two saw the old blob with a zero offset and rejected every id
    in the just-completed blob. With ENABLE_FAST_SDK=OFF that reaches
    _TSReleaseAssert through TSStatInt*, so it aborts rather than losing a
    count. Reversing the stores only trades it for accepting ids in a blob
    nothing has been written to; two atomics have no coherent pair either way.
    
    One atomic holding the blob index above the offset, packed as an id is
    packed, fixes it: crossing a blob is a single release store. The value
    only ever increases, so an id is allocated exactly when it packs below
    the bound, which reduces the gate on every id based accessor to one
    acquire load and one compare. Acquiring the bound also acquires the blob
    install, so the null blob check goes away. It is also the id of the next
    free slot, which is what iteration wants for its end bound.
    
    Drop createSpan with it. It has no callers outside the tests, and it was
    the only path that could leave a blob partly filled -- it skipped to a
    fresh blob when a span did not fit, abandoning tail slots that were never
    handed out and that the packed bound would count as allocated. Without
    it, blobs fill contiguously and "packs below the bound" means exactly
    "was handed out".
    
    * Make the concurrent lookup test check what it claims
    
    Two ways it could pass without testing anything. Readers only published
    their tally on exit and nothing made them run before the writer finished,
    so on one CPU every reader could see stop and resolve nothing while
    resolved > 0 still held; it now publishes each resolution as it happens
    and the writer waits for one before stopping. And an id that lookup()
    clamps resolves to the reserved bad_id slot, whose name is not empty, so
    the name check could not detect a clamp; it now compares against the name
    that id must have.
    
    * Include <limits> and stop binding an unused offset
    
    Dropping createSpan took swoc/MemSpan.h with it, and that was what
    supplied <limits> for NOT_FOUND's numeric_limits. The header still
    compiles, through some other transitive path, which is exactly what makes
    it worth declaring.
    
    addBlob() destructured the packed value but only ever used the blob half.
    
    * Take the lock before touching a slot's name in rename()
    
    The name is the key _lookups is indexed by, so replacing it belongs
    entirely inside the lock. Nothing read the string outside it before --
    binding a reference to it does not touch its bytes -- but computing that
    reference outside the lock made the boundary look wider than it is, and
    there is no reason for anything here to sit outside.
    
    * Metrics: trim comments, and drop ones about a check that is gone
    
    Two comments in the malformed-offset test explained that the null blob
    check, not the offset test, would reject those ids with only one blob
    allocated. The packed bound removed that check, so the reasoning no
    longer applied.
    
    * Remove rename()
    
    It mutated a slot's name while name() and lookup(id, &out_name) read that
    same std::string without the mutex and hand out views into it, which
    moonchen reproduced as a TSAN race. Locking rename() does not fix it; the
    readers are the lock free paths this PR exists to keep. Giving names
    immutable storage with its own lifetime rules would, but nothing outside
    the tests calls rename().
    
    Without it a name is written once before the store that publishes it and
    never changes, so those readers are correct by construction.
---
 include/tsutil/Metrics.h                   | 107 ++++++++--------
 src/records/unit_tests/test_RecRegister.cc |   4 +-
 src/tsutil/Metrics.cc                      | 129 ++++++--------------
 src/tsutil/unit_tests/test_Metrics.cc      | 187 +++++++++++++++++++---------
 tools/benchmark/CMakeLists.txt             |   3 +
 tools/benchmark/benchmark_Metrics.cc       | 190 +++++++++++++++++++++++++++++
 6 files changed, 416 insertions(+), 204 deletions(-)

diff --git a/include/tsutil/Metrics.h b/include/tsutil/Metrics.h
index 40cdef9522..741b1e069a 100644
--- a/include/tsutil/Metrics.h
+++ b/include/tsutil/Metrics.h
@@ -30,13 +30,12 @@
 #include <mutex>
 #include <atomic>
 #include <cstdint>
+#include <limits>
 #include <string>
 #include <string_view>
 #include <variant>
 #include <optional>
 
-#include "swoc/MemSpan.h"
-
 #include "tsutil/Assert.h"
 
 namespace ts
@@ -85,8 +84,7 @@ public:
 
   enum class MetricType : int { COUNTER = 0, GAUGE };
 
-  using IdType   = int32_t; // Could be a tuple, but one way or another, they have to be combined to an int32_t.
-  using SpanType = swoc::MemSpan<AtomicType>;
+  using IdType = int32_t; // Could be a tuple, but one way or another, they have to be combined to an int32_t.
 
   static constexpr uint16_t MAX_BLOBS        = 8192;
   static constexpr uint16_t MAX_SIZE         = 1024;                               // For a total of 8M metrics
@@ -147,12 +145,6 @@ public:
   {
     return _storage->lookup(id, out_name, type);
   }
-  bool
-  rename(IdType id, const std::string_view name)
-  {
-    return _storage->rename(id, name);
-  }
-
   AtomicType &
   operator[](IdType id)
   {
@@ -267,9 +259,7 @@ public:
   iterator
   end() const
   {
-    auto [blob, offset] = _storage->current();
-
-    return iterator(*this, _makeId(blob, offset, MetricType::COUNTER));
+    return iterator(*this, _storage->next_free_id());
   }
 
   iterator
@@ -292,12 +282,6 @@ private:
     return _storage->create(name, type);
   }
 
-  SpanType
-  _createSpan(size_t size, MetricType type, IdType *id = nullptr)
-  {
-    return _storage->createSpan(size, type, id);
-  }
-
   // These are little helpers around managing the ID's
   static constexpr std::tuple<uint16_t, uint16_t>
   _splitID(IdType value)
@@ -308,7 +292,7 @@ private:
   static constexpr MetricType
   _extractType(IdType value)
   {
-    return MetricType{value >> METRIC_TYPE_BITS};
+    return MetricType{static_cast<int>((static_cast<uint32_t>(value) >> METRIC_TYPE_BITS) & 0x1)};
   }
 
   static constexpr IdType
@@ -318,13 +302,31 @@ private:
     return (t << METRIC_TYPE_BITS | blob << 16 | offset);
   }
 
+  /// As @c _makeId, without the type bits.
+  static constexpr uint32_t
+  _pack(uint16_t blob, uint16_t offset)
+  {
+    return static_cast<uint32_t>(blob) << 16 | offset;
+  }
+
+  // A packed position must not reach the type bits, and an offset must fit its field.
+  static_assert(MAX_SIZE <= 0x10000);
+  static_assert(MAX_BLOBS <= (1 << (METRIC_TYPE_BITS - 16)));
+
   class Storage
   {
-    BlobStorage        _blobs;
-    uint16_t           _cur_blob = 0;
-    uint16_t           _cur_off  = 0;
-    LookupTable        _lookups;
-    mutable std::mutex _mutex;
+    /* The next free slot, packed as @c _makeId packs one. A single value because a reader that
+     * caught a new offset against an old blob index, or the reverse, would reject ids that exist
+     * or accept ids that do not. Release stored last, after the blob pointer or the slot's name it
+     * publishes. Only ever increases, so an id is allocated exactly when it packs below it.
+     *
+     * A slot's name is written once, before the store that publishes it, and never changes, which
+     * is what lets @c name and @c lookup hand out a view of it without the mutex.
+     */
+    BlobStorage           _blobs;
+    std::atomic<uint32_t> _next_free{0};
+    LookupTable           _lookups;
+    mutable std::mutex    _mutex;
 
   public:
     Storage(const Storage &)            = delete;
@@ -347,22 +349,43 @@ private:
     AtomicType      *lookup(Metrics::IdType id, std::string_view *out_name = nullptr, MetricType *out_type = nullptr) const;
     std::string_view name(IdType id) const;
     MetricType       type(IdType id) const;
-    SpanType         createSpan(size_t size, const MetricType type = MetricType::COUNTER, IdType *id = nullptr);
-    bool             rename(IdType id, const std::string_view name);
 
-    std::pair<int16_t, int16_t>
-    current() const
+    /// The id the next slot will get, which is also iteration's exclusive bound.
+    IdType
+    next_free_id() const
     {
-      std::lock_guard lock(_mutex);
-      return {_cur_blob, _cur_off};
+      return static_cast<IdType>(_next_free.load(std::memory_order_acquire));
     }
 
     bool
     valid(IdType id) const
     {
-      auto [blob, entry] = _splitID(id);
+      return _is_allocated(id);
+    }
+
+  private:
+    /** Whether @a id names an allocated slot.
+     *
+     * The gate for every id based accessor, since ids from the @c TSStat* API are untrusted. An id
+     * qualifies when it is non-negative, its offset is one @c _makeId could produce, and its slot
+     * has been handed out.
+     */
+    bool
+    _is_allocated(IdType id) const
+    {
+      if (id < 0) {
+        return false;
+      }
+
+      auto [blob_ix, offset] = _splitID(id);
 
-      return (id >= 0 && ((blob < _cur_blob && entry < MAX_SIZE) || (blob == _cur_blob && entry <= _cur_off)));
+      // Not implied below: an earlier blob can name an offset past MAX_SIZE and still pack under.
+      if (offset >= MAX_SIZE) {
+        return false;
+      }
+
+      // Acquiring the bound acquires the blob install, so _blobs needs no check of its own.
+      return _pack(blob_ix, offset) < _next_free.load(std::memory_order_acquire);
     }
   };
 
@@ -376,7 +399,6 @@ public:
   {
   public:
     using self_type = Gauge;
-    using SpanType  = Metrics::SpanType;
 
     class AtomicType : public Metrics::AtomicType
     {
@@ -452,14 +474,6 @@ public:
       return reinterpret_cast<AtomicType *>(instance.lookup(instance._create(tmpname, MetricType::GAUGE)));
     }
 
-    static Metrics::Gauge::SpanType
-    createSpan(size_t size, IdType *id = nullptr)
-    {
-      auto &instance = Metrics::instance();
-
-      return instance._createSpan(size, MetricType::GAUGE, id);
-    }
-
     static void
     increment(AtomicType *metric, uint64_t val = 1)
     {
@@ -494,7 +508,6 @@ public:
   {
   public:
     using self_type = Counter;
-    using SpanType  = Metrics::SpanType;
 
     class AtomicType : public Metrics::AtomicType
     {
@@ -570,14 +583,6 @@ public:
       return reinterpret_cast<AtomicType *>(instance.lookup(instance._create(tmpname, MetricType::COUNTER)));
     }
 
-    static Metrics::Counter::SpanType
-    createSpan(size_t size, IdType *id = nullptr)
-    {
-      auto &instance = Metrics::instance();
-
-      return instance._createSpan(size, MetricType::COUNTER, id);
-    }
-
     static void
     increment(AtomicType *metric, uint64_t val = 1)
     {
diff --git a/src/records/unit_tests/test_RecRegister.cc b/src/records/unit_tests/test_RecRegister.cc
index 38fb1adbf0..a406b6fbd6 100644
--- a/src/records/unit_tests/test_RecRegister.cc
+++ b/src/records/unit_tests/test_RecRegister.cc
@@ -26,6 +26,7 @@
 #include "test_Diags.h"
 
 #include <atomic>
+#include <string>
 #include <thread>
 
 TEST_CASE("RecRegisterConfig - Type Dispatch", "[librecords][RecConfig]")
@@ -103,7 +104,8 @@ TEST_CASE("RecLookupRecord - Concurrent metric registration", "[librecords][RecL
   std::atomic<bool> finished{false};
   std::thread       register_metrics([&]() {
     for (int i = 0; i < 100000; ++i) {
-      ts::Metrics::Counter::createSpan(1);
+      // Any registration will do; the point is to grow the store while lookups run.
+      ts::Metrics::Counter::create("proxy.test.concurrent.reg." + std::to_string(i));
     }
     finished.store(true, std::memory_order_release);
   });
diff --git a/src/tsutil/Metrics.cc b/src/tsutil/Metrics.cc
index 110b0ca0d8..0b9ef8dc5a 100644
--- a/src/tsutil/Metrics.cc
+++ b/src/tsutil/Metrics.cc
@@ -58,12 +58,16 @@ Metrics::Storage::addBlob() // The mutex must be held before calling this!
 {
   auto blob = std::make_unique<Metrics::NamesAndAtomics>();
 
+  auto const cur_blob = static_cast<uint16_t>(_next_free.load(std::memory_order_relaxed) >> 16);
+
   debug_assert(blob);
-  // The write below is to _blobs[_cur_blob + 1], so the last usable blob index is MAX_BLOBS - 1.
-  release_assert(_cur_blob < MAX_BLOBS - 1);
+  // The write below is to _blobs[cur_blob + 1], so the last usable blob index is MAX_BLOBS - 1.
+  release_assert(cur_blob < MAX_BLOBS - 1);
+
+  _blobs[cur_blob + 1] = std::move(blob);
 
-  _blobs[++_cur_blob] = std::move(blob);
-  _cur_off            = 0;
+  // One store publishes both; the install above is sequenced before it.
+  _next_free.store(_pack(cur_blob + 1, 0), std::memory_order_release);
 }
 
 Metrics::IdType
@@ -76,22 +80,27 @@ Metrics::Storage::create(std::string_view name, const MetricType type)
     return it->second;
   }
 
-  // The slot is written below and the bookkeeping only then advances, calling addBlob() once
-  // _cur_off reaches MAX_SIZE. Refusing the final slot of the final blob keeps addBlob() from
-  // ever being reached in an exhausted store, at a cost of one slot out of MAX_BLOBS * MAX_SIZE.
-  if (_cur_blob >= MAX_BLOBS - 1 && _cur_off >= MAX_SIZE - 1) {
+  // The slot is written below and the bookkeeping only then advances, calling addBlob() once the
+  // offset reaches MAX_SIZE. Refusing the final slot of the final blob keeps addBlob() from ever
+  // being reached in an exhausted store, at a cost of one slot out of MAX_BLOBS * MAX_SIZE.
+  auto const [cur_blob, cur_off] = _splitID(static_cast<IdType>(_next_free.load(std::memory_order_relaxed)));
+
+  if (cur_blob >= MAX_BLOBS - 1 && cur_off >= MAX_SIZE - 1) {
     return 0; // Slot 0 is the reserved bad_id. Cannot grow further.
   }
 
-  Metrics::IdType           id    = _makeId(_cur_blob, _cur_off, type);
-  Metrics::NamesAndAtomics *blob  = _blobs[_cur_blob].get();
+  Metrics::IdType           id    = _makeId(cur_blob, cur_off, type);
+  Metrics::NamesAndAtomics *blob  = _blobs[cur_blob].get();
   Metrics::NameStorage     &names = std::get<0>(*blob);
 
-  names[_cur_off] = std::make_tuple(std::string(name), id);
-  _lookups.emplace(std::get<0>(names[_cur_off]), id);
+  names[cur_off] = std::make_tuple(std::string(name), id);
+  _lookups.emplace(std::get<0>(names[cur_off]), id);
 
-  if (++_cur_off >= MAX_SIZE) {
-    addBlob(); // This resets _cur_off to 0 as well
+  if (cur_off + 1 >= MAX_SIZE) {
+    addBlob();
+  } else {
+    // Publishes the slot's name.
+    _next_free.store(_pack(cur_blob, cur_off + 1), std::memory_order_release);
   }
 
   return id;
@@ -113,15 +122,16 @@ Metrics::Storage::lookup(const std::string_view name) const
 Metrics::AtomicType *
 Metrics::Storage::lookup(Metrics::IdType id, std::string_view *out_name, Metrics::MetricType *out_type) const
 {
-  auto [blob_ix, offset]         = _splitID(id);
-  Metrics::NamesAndAtomics *blob = _blobs[blob_ix].get();
+  auto [blob_ix, offset] = _splitID(id);
 
-  // Do a sanity check on the ID, to make sure we don't index outside of the realm of possibility.
-  if (!blob || (blob_ix == _cur_blob && offset > _cur_off)) {
-    blob   = _blobs[0].get();
-    offset = 0;
+  // Anything not naming an allocated slot resolves to the reserved bad_id slot.
+  if (!_is_allocated(id)) {
+    blob_ix = 0;
+    offset  = 0;
   }
 
+  Metrics::NamesAndAtomics *blob = _blobs[blob_ix].get();
+
   if (out_name) {
     *out_name = std::get<0>(std::get<0>(*blob)[offset]);
   }
@@ -159,15 +169,16 @@ Metrics::Storage::lookup(const std::string_view name, Metrics::IdType *out_id, M
 std::string_view
 Metrics::Storage::name(Metrics::IdType id) const
 {
-  auto [blob_ix, offset]         = _splitID(id);
-  Metrics::NamesAndAtomics *blob = _blobs[blob_ix].get();
+  auto [blob_ix, offset] = _splitID(id);
 
-  // Do a sanity check on the ID, to make sure we don't index outside of the realm of possibility.
-  if (!blob || (blob_ix == _cur_blob && offset > _cur_off)) {
-    blob   = _blobs[0].get();
-    offset = 0;
+  // Anything not naming an allocated slot resolves to the reserved bad_id slot.
+  if (!_is_allocated(id)) {
+    blob_ix = 0;
+    offset  = 0;
   }
 
+  Metrics::NamesAndAtomics *blob = _blobs[blob_ix].get();
+
   const std::string &result = std::get<0>(std::get<0>(*blob)[offset]);
 
   return result;
@@ -179,72 +190,6 @@ Metrics::Storage::type(IdType id) const
   return _extractType(id);
 }
 
-Metrics::SpanType
-Metrics::Storage::createSpan(size_t size, Metrics::MetricType type, Metrics::IdType *id)
-{
-  release_assert(size <= MAX_SIZE);
-  std::lock_guard lock(_mutex);
-
-  // On the final blob there is nowhere left to grow, so refuse a span that would fill or overflow
-  // it rather than letting addBlob() assert. Same intent as the guard in create(), and the same
-  // cost: some slots of the last blob go unused.
-  if (_cur_blob >= MAX_BLOBS - 1 && _cur_off + size >= MAX_SIZE) {
-    if (id) {
-      *id = 0; // Slot 0 is the reserved bad_id.
-    }
-    return {};
-  }
-
-  // A span has to be contiguous, so one that does not fit in the current blob starts a new one.
-  if (_cur_off + size > MAX_SIZE) {
-    addBlob();
-  }
-
-  Metrics::IdType           span_start = _makeId(_cur_blob, _cur_off, type);
-  Metrics::NamesAndAtomics *blob       = _blobs[_cur_blob].get();
-  Metrics::AtomicStorage   &atomics    = std::get<1>(*blob);
-  Metrics::SpanType         span       = Metrics::SpanType(&atomics[_cur_off], size);
-
-  if (id) {
-    *id = span_start;
-  }
-
-  _cur_off += size;
-
-  // create() grows as soon as it consumes the last slot; do the same here. Otherwise a span ending
-  // exactly on the boundary leaves _cur_off at MAX_SIZE, and the next create() writes one past the
-  // end of the blob's name array. It also makes end() unreachable for iterator::next(), which
-  // wraps on ++offset == MAX_SIZE.
-  if (_cur_off >= MAX_SIZE) {
-    addBlob();
-  }
-
-  return span;
-}
-
-bool
-Metrics::Storage::rename(Metrics::IdType id, std::string_view name)
-{
-  auto [blob_ix, offset]         = _splitID(id);
-  Metrics::NamesAndAtomics *blob = _blobs[blob_ix].get();
-
-  // We can only rename Metrics that are already allocated
-  if (!blob || (blob_ix == _cur_blob && offset > _cur_off)) {
-    return false;
-  }
-
-  std::string    &cur = std::get<0>(std::get<0>(*blob)[offset]);
-  std::lock_guard lock(_mutex);
-
-  if (cur.length() > 0) {
-    _lookups.erase(cur);
-  }
-  cur = name;
-  _lookups.emplace(cur, id);
-
-  return true;
-}
-
 // Iterator implementation
 void
 Metrics::iterator::next()
diff --git a/src/tsutil/unit_tests/test_Metrics.cc b/src/tsutil/unit_tests/test_Metrics.cc
index 960324997b..f267097807 100644
--- a/src/tsutil/unit_tests/test_Metrics.cc
+++ b/src/tsutil/unit_tests/test_Metrics.cc
@@ -24,8 +24,10 @@
 #include <catch2/catch_test_macros.hpp>
 
 #include <algorithm>
+#include <atomic>
 #include <array>
 #include <iterator>
+#include <limits>
 #include <memory>
 #include <string>
 #include <thread>
@@ -93,37 +95,6 @@ TEST_CASE("Metrics", "[libtsapi][Metrics]")
     REQUIRE(m[storeid].load() == 42);
   }
 
-  SECTION("Span allocation")
-  {
-    ts::Metrics::IdType span_id;
-    auto                fooid = m.lookup("foo");
-    auto                span  = Metrics::Counter::createSpan(17, &span_id);
-
-    REQUIRE(span.size() == 17);
-    // Not fixed offsets: those only hold against a virgin store. Assert instead that the span
-    // was allocated above the earlier metric and that every id in it is valid. Both ids are
-    // counters, so they are directly comparable -- ids encode the metric type, and so are not
-    // ordered across differing types.
-    REQUIRE(fooid != ts::Metrics::NOT_FOUND);
-    REQUIRE(span_id != ts::Metrics::NOT_FOUND);
-    REQUIRE(span_id > fooid);
-    for (size_t i = 0; i < span.size(); ++i) {
-      REQUIRE(m.valid(span_id + static_cast<ts::Metrics::IdType>(i)));
-    }
-
-    m.rename(span_id + 0, "span.0");
-    m.rename(span_id + 1, "span.1");
-    m.rename(span_id + 2, "span.2");
-    REQUIRE(m.name(fooid) == "foo");
-    REQUIRE(m.name(span_id + 0) == "span.0");
-    REQUIRE(m.name(span_id + 1) == "span.1");
-    REQUIRE(m.name(span_id + 2) == "span.2");
-    m.rename(fooid, "foo-new");
-    REQUIRE(m.name(fooid) == "foo-new");
-    REQUIRE(m.lookup("foo") == ts::Metrics::NOT_FOUND);
-    REQUIRE(m.lookup("foo-new") == fooid);
-  }
-
   SECTION("lookup")
   {
     auto nm = m.lookup("notametric");
@@ -608,35 +579,131 @@ TEST_CASE("Metrics blob growth boundary", "[libtsapi][Metrics]")
   REQUIRE(std::adjacent_find(sorted_ptrs.begin(), sorted_ptrs.end()) == sorted_ptrs.end());
 }
 
-TEST_CASE("Metrics span lands exactly on a blob boundary", "[libtsapi][Metrics]")
+TEST_CASE("Metrics malformed id offsets resolve to bad_id", "[libtsapi][Metrics]")
+{
+  // An id's offset field is 16 bits but a real offset is below MAX_SIZE, so a malformed one must
+  // not index past a blob's arrays. Filling past one blob puts the ids below in earlier blobs,
+  // where they pack under the bound and only the MAX_SIZE test rejects them.
+  auto &h = Metrics::hidden_instance();
+
+  for (int i = 0; i < Metrics::MAX_SIZE + 8; ++i) {
+    REQUIRE(Metrics::Counter::createHiddenPtr("f1.fill." + std::to_string(i)) != nullptr);
+  }
+
+  auto const *bad = h.lookup(Metrics::IdType{0}); // the reserved bad_id slot
+  REQUIRE(bad != nullptr);
+
+  for (Metrics::IdType id : {Metrics::IdType{0x0000FFFF}, Metrics::IdType{0x00000400}, Metrics::IdType{0x0001FFFF}}) {
+    REQUIRE(h.valid(id) == false);
+    REQUIRE(h.lookup(id) == bad);
+    REQUIRE(h.name(id) == h.name(Metrics::IdType{0}));
+  }
+}
+
+TEST_CASE("Metrics id lookup is safe against concurrent creation", "[libtsapi][Metrics]")
 {
-  // A span has to be contiguous, so createSpan(MAX_SIZE) always starts a fresh blob and then fills
-  // it completely, whatever the current offset was. That makes this the one span size that reaches
-  // the boundary case deterministically: the offset ends up at MAX_SIZE, and unlike create(),
-  // createSpan used not to grow a new blob afterwards. The next create() then indexed one past the
-  // end of the blob's name array, and end() became an id that iterator::next() can never reach
-  // because it wraps at ++offset == MAX_SIZE.
+  // The id based read paths take no lock, so resolving an id races a concurrent create. Run both
+  // sides at once, over enough metrics to cross several blob boundaries. Under the tsan preset a
+  // non-atomic allocation counter reports a data race here; relaxing the memory orders does not,
+  // since atomics are race free at any ordering.
   //
-  // createSpan only ever targets the published store, so this necessarily allocates there.
-  Metrics::IdType span_id = Metrics::NOT_FOUND;
-  auto            span    = Metrics::Counter::createSpan(Metrics::MAX_SIZE, &span_id);
-
-  REQUIRE(span.size() == Metrics::MAX_SIZE);
-  REQUIRE(span_id != Metrics::NOT_FOUND);
-  REQUIRE(span_id != 0); // 0 is the reserved bad_id, returned only when the store cannot grow.
-
-  // The store must still be usable, and the new metric must be a real, resolvable entry rather
-  // than something written past the end of a blob.
-  auto p = Metrics::Counter::createPtr("span.boundary.after");
-  REQUIRE(p != nullptr);
-
-  auto &m  = Metrics::instance();
-  auto  id = m.lookup("span.boundary.after");
-  REQUIRE(id != Metrics::NOT_FOUND);
-  REQUIRE(m.valid(id));
-
-  // And it must behave like any other metric.
-  Metrics::Counter::increment(p, 7);
-  REQUIRE(Metrics::Counter::load(p) == 7);
-  REQUIRE(Metrics::Counter::createPtr("span.boundary.after") == p);
+  // Readers take ids from what the writer has registered rather than counting integers: an id packs
+  // the blob index above the offset, so consecutive integers only ever name the first blob. The
+  // store is relaxed, so a reader can pick up an id whose slot is not published yet, which is the
+  // case of interest.
+  constexpr int N_READERS = 4;
+  constexpr int N_CREATE  = Metrics::MAX_SIZE * 2 + 64;
+
+  auto             &h = Metrics::hidden_instance();
+  std::atomic<bool> stop{false};
+  std::atomic<int>  ready{0};
+  std::atomic<int>  mismatches{0};
+  std::atomic<int>  resolved{0};
+
+  std::vector<std::atomic<Metrics::IdType>> created(N_CREATE);
+
+  for (auto &c : created) {
+    c.store(Metrics::NOT_FOUND, std::memory_order_relaxed);
+  }
+
+  // A clamped lookup resolves to the bad_id slot, whose name is not empty, so only the expected
+  // name detects one.
+  std::vector<std::string> names;
+
+  names.reserve(N_CREATE);
+  for (int i = 0; i < N_CREATE; ++i) {
+    names.push_back("pub.order." + std::to_string(i));
+  }
+
+  std::vector<std::thread> readers;
+
+  for (int t = 0; t < N_READERS; ++t) {
+    readers.emplace_back([&]() {
+      ready.fetch_add(1, std::memory_order_release);
+
+      while (!stop.load(std::memory_order_relaxed)) {
+        for (int i = 0; i < N_CREATE; ++i) {
+          auto const id = created[i].load(std::memory_order_relaxed);
+
+          if (id == Metrics::NOT_FOUND || !h.valid(id)) {
+            continue;
+          }
+
+          // valid() accepted the id, so lookup() must return that metric and not clamp.
+          std::string_view    name;
+          Metrics::MetricType type;
+          auto               *m = h.lookup(id, &name, &type);
+
+          if (m == nullptr || name != names[i]) {
+            mismatches.fetch_add(1, std::memory_order_relaxed);
+          }
+
+          // Published as it happens so the writer can wait for one.
+          resolved.fetch_add(1, std::memory_order_relaxed);
+        }
+      }
+    });
+  }
+
+  // Readers must be in their loops before the writer starts, or nothing races.
+  while (ready.load(std::memory_order_acquire) < N_READERS) {
+    std::this_thread::yield();
+  }
+
+  for (int i = 0; i < N_CREATE; ++i) {
+    REQUIRE(Metrics::Counter::createHiddenPtr(names[i]) != nullptr);
+    created[i].store(h.lookup(names[i]), std::memory_order_relaxed);
+  }
+
+  // Being in the loop is not doing work: on one CPU the writer can finish first and every reader
+  // would then see stop. Wait for a real resolution so the check below cannot pass vacuously.
+  while (resolved.load(std::memory_order_relaxed) == 0) {
+    std::this_thread::yield();
+  }
+
+  stop.store(true, std::memory_order_relaxed);
+  for (auto &r : readers) {
+    r.join();
+  }
+
+  CHECK(mismatches.load() == 0);
+  CHECK(resolved.load() > 0);
+
+  auto lo = std::numeric_limits<Metrics::IdType>::max();
+  auto hi = std::numeric_limits<Metrics::IdType>::min();
+
+  for (int i = 0; i < N_CREATE; ++i) {
+    auto const id = h.lookup(names[i]);
+
+    REQUIRE(id != Metrics::NOT_FOUND);
+    REQUIRE(h.valid(id));
+    REQUIRE(h.name(id) == names[i]);
+
+    lo = std::min(lo, id);
+    hi = std::max(hi, id);
+  }
+
+  // More metrics than fit in one blob, so the ids must span blobs. A spread no wider than a blob
+  // would mean the sweep above never left the first one.
+  REQUIRE(hi - lo > Metrics::MAX_SIZE);
 }
diff --git a/tools/benchmark/CMakeLists.txt b/tools/benchmark/CMakeLists.txt
index 7d11b3eae9..1456824fb3 100644
--- a/tools/benchmark/CMakeLists.txt
+++ b/tools/benchmark/CMakeLists.txt
@@ -33,6 +33,9 @@ target_link_libraries(benchmark_ProxyAllocator PRIVATE Catch2::Catch2WithMain ts
 add_executable(benchmark_SharedMutex benchmark_SharedMutex.cc)
 target_link_libraries(benchmark_SharedMutex PRIVATE Catch2::Catch2 ts::tscore libswoc::libswoc)
 
+add_executable(benchmark_Metrics benchmark_Metrics.cc)
+target_link_libraries(benchmark_Metrics PRIVATE Catch2::Catch2 ts::tsutil libswoc::libswoc)
+
 add_executable(benchmark_Random benchmark_Random.cc)
 target_link_libraries(benchmark_Random PRIVATE Catch2::Catch2WithMain ts::tscore)
 
diff --git a/tools/benchmark/benchmark_Metrics.cc b/tools/benchmark/benchmark_Metrics.cc
new file mode 100644
index 0000000000..e42b092a7f
--- /dev/null
+++ b/tools/benchmark/benchmark_Metrics.cc
@@ -0,0 +1,190 @@
+/** @file
+
+  Micro benchmark tool for ts::Metrics
+
+  Metric values are lock free atomics; reaching one from an id is not. Four cases, scaled by thread
+  count:
+
+    increment(id)     valid() then lookup(id) then fetch_add, as TSStatIntIncrement does
+    increment(ptr)    a bare fetch_add on a cached pointer, as core does
+    lookup(id)        lock free id resolution
+    lookup(name)      the same resolution through the mutex guarded name map
+
+  increment(id) against increment(ptr) is what an id costs a plugin. lookup(id) against
+  lookup(name) isolates the mutex, and serves as a control: it must degrade with thread count, or
+  the harness is not loading the machine.
+
+  @section license License
+
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+ */
+
+#define CATCH_CONFIG_ENABLE_BENCHMARKING
+
+#include <catch2/catch_test_macros.hpp>
+#include <catch2/catch_session.hpp>
+#include <catch2/benchmark/catch_benchmark.hpp>
+
+#include "tsutil/Metrics.h"
+
+#include <atomic>
+#include <string>
+#include <thread>
+#include <vector>
+
+using ts::Metrics;
+
+namespace
+{
+// Args
+struct Conf {
+  int nthreads = 1;
+  int nops     = 1000;
+  int nmetrics = 64;
+};
+
+Conf conf;
+
+/// The metrics every case operates on, created once.
+struct Fixture {
+  std::vector<Metrics::IdType>                ids;
+  std::vector<Metrics::Counter::AtomicType *> ptrs;
+  std::vector<std::string>                    names;
+
+  Fixture()
+  {
+    auto &m = Metrics::instance();
+
+    ids.reserve(conf.nmetrics);
+    ptrs.reserve(conf.nmetrics);
+    names.reserve(conf.nmetrics);
+
+    for (int i = 0; i < conf.nmetrics; ++i) {
+      names.push_back("benchmark.metrics." + std::to_string(i));
+
+      // Registers the name too, so the id and name lookups resolve to the same metric.
+      ptrs.push_back(Metrics::Counter::createPtr(names.back()));
+      ids.push_back(m.lookup(names.back()));
+    }
+  }
+};
+
+Fixture *fixture = nullptr;
+
+/// Run @a op on every thread, @c nops times each. The return value only defeats optimization.
+template <typename F>
+int64_t
+run(F &&op)
+{
+  std::vector<std::thread> threads;
+  std::atomic<int64_t>     sink{0};
+
+  threads.reserve(conf.nthreads);
+
+  for (int t = 0; t < conf.nthreads; ++t) {
+    threads.emplace_back([t, &sink, &op]() {
+      int64_t local = 0;
+
+      for (int i = 0; i < conf.nops; ++i) {
+        // Stride per thread, or this measures cacheline ping-pong on one atomic.
+        local += op((t + i) % conf.nmetrics);
+      }
+      sink.fetch_add(local, std::memory_order_relaxed);
+    });
+  }
+
+  for (auto &th : threads) {
+    th.join();
+  }
+
+  return sink.load();
+}
+
+} // namespace
+
+TEST_CASE("Micro benchmark of ts::Metrics", "")
+{
+  auto &m = Metrics::instance();
+
+  SECTION("increment by id")
+  {
+    BENCHMARK("increment(id)")
+    {
+      return run([&m](int i) -> int64_t {
+        auto id = fixture->ids[i];
+
+        return m.valid(id) ? m.increment(id, 1) : 0;
+      });
+    };
+  }
+
+  SECTION("increment by cached pointer")
+  {
+    // The floor: no resolution at all.
+    BENCHMARK("increment(ptr)")
+    {
+      return run([](int i) -> int64_t {
+        Metrics::Counter::increment(fixture->ptrs[i], 1);
+
+        return 1;
+      });
+    };
+  }
+
+  SECTION("lookup by id")
+  {
+    BENCHMARK("lookup(id)")
+    {
+      return run([&m](int i) -> int64_t { return m.lookup(fixture->ids[i]) != nullptr; });
+    };
+  }
+
+  SECTION("lookup by name")
+  {
+    BENCHMARK("lookup(name)")
+    {
+      return run([&m](int i) -> int64_t { return m.lookup(fixture->names[i]) != Metrics::NOT_FOUND; });
+    };
+  }
+}
+
+int
+main(int argc, char *argv[])
+{
+  Catch::Session session;
+
+  using namespace Catch::Clara;
+
+  // clang-format off
+  auto cli = session.cli() |
+    Opt(conf.nthreads, "")["--ts-nthreads"]("number of threads (default: 1)") |
+    Opt(conf.nops, "")["--ts-nops"]("operations per thread per run (default: 1000)") |
+    Opt(conf.nmetrics, "")["--ts-nmetrics"]("distinct metrics to spread across (default: 64)");
+  // clang-format on
+
+  session.cli(cli);
+
+  int returnCode = session.applyCommandLine(argc, argv);
+  if (returnCode != 0) {
+    return returnCode;
+  }
+
+  Fixture f;
+  fixture = &f;
+
+  return session.run();
+}