You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2026/08/28 13:02:18 UTC

Re: [PR] Scalable Topics: `pulsar::st` queue consumer — per-segment fan-in over a mux receive queue [pulsar-client-cpp]

lhotari commented on code in PR #605:
URL: https://github.com/apache/pulsar-client-cpp/pull/605#discussion_r3880773153


##########
lib/ConsumerImpl.cc:
##########
@@ -823,6 +823,24 @@ void ConsumerImpl::activeConsumerChanged(bool isActive) {
     }
 }
 
+void ConsumerImpl::reachedEndOfTopic() {
+    hasReachedEndOfTopic_ = true;

Review Comment:
   **[BUG] hasReachedEndOfTopic_ is never cleared, so a reconnect can report a stale terminal result and permanently drop a segment**
   
   This flag is set once and never reset — it is the only write in the file. Meanwhile the reconnect path clears the prefetch queue (`clearReceiveQueue()` at line 244, `incomingMessages_.clear()` at line 337) and then sends fresh flow permits.
   
   That leaves a window after every reconnect in which `incomingMessages_` is empty but redelivery of unacked messages has not yet arrived. A `receiveAsync` landing in that window falls through to the new branch at line 1210 and returns `ResultTopicTerminated` even though messages are still owed.
   
   Termination means no *new* publications; it does not mean already-delivered-but-unacked entries stop needing redelivery.
   
   Failure scenario: a segment consumer with unacked messages loses its connection and re-subscribes. Before the broker redelivers, the st queue consumer's receive loop calls `receiveAsync`, gets the stale `ResultTopicTerminated`, and takes the drain path in `QueueConsumerImpl.cc:203-213` — closing the consumer and recording the segment in `drainedSegments_`. Those messages are never delivered to the application for the life of this consumer.
   
   Clearing the flag in `connectionOpened()`/`handleCreateConsumer()` alongside the queue clear would close this.



##########
lib/ConsumerImpl.cc:
##########
@@ -1189,6 +1207,14 @@ void ConsumerImpl::receiveAsync(const ReceiveCallback& callback) {
         messageProcessed(msg);
         msg = interceptors_->beforeConsume(Consumer(shared_from_this()), msg);
         callback(ResultOk, msg);
+    } else if (hasReachedEndOfTopic_) {
+        // Terminated topic with nothing left buffered: fail the receive rather than parking it
+        // forever waiting for a message that will never arrive.

Review Comment:
   **[BUG] Blocking receive() still waits forever on a drained terminated topic**
   
   The new termination check exists only on the async path. The untimed `receiveHelper(Message&)` (line 1246) blocks in `incomingMessages_.pop(msg)` and never consults `hasReachedEndOfTopic_`, and `reachedEndOfTopic()` only completes `pendingReceives_`, which synchronous receives do not use. The timed overload returns `ResultTimeout` rather than `ResultTopicTerminated`.
   
   Failure scenario: an application calls `consumer.receive(msg)` on a topic that is terminated and fully drained. It blocks forever. Before this PR the same call also hung, but the connection churn made it visible; now the client handles the command cleanly and the hang is silent.
   
   Not a regression, but the capability is only half-wired — worth finishing here so the sync and async paths agree.



##########
lib/st/QueueConsumerImpl.cc:
##########
@@ -0,0 +1,314 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "QueueConsumerImpl.h"
+
+#include <cstdint>
+#include <string>
+#include <unordered_set>
+#include <utility>
+#include <variant>
+#include <vector>
+
+#include "MessageIdImpl.h"
+#include "MessageImpl.h"
+#include "lib/LogUtils.h"
+
+DECLARE_LOG_OBJECT()
+
+namespace pulsar::st {
+
+namespace {
+
+pulsar::InitialPosition toClassicInitialPosition(SubscriptionInitialPosition position) {
+    return position == SubscriptionInitialPosition::Earliest ? pulsar::InitialPositionEarliest
+                                                             : pulsar::InitialPositionLatest;
+}
+
+}  // namespace
+
+QueueConsumerImpl::QueueConsumerImpl(pulsar::ClientImplPtr classic, QueueConsumerConfig config)
+    : classic_(std::move(classic)),
+      config_(std::move(config)),
+      topic_(config_.topic),
+      subscription_(config_.subscriptionName),
+      consumerName_(config_.consumerName.value_or(std::string{})),
+      receiveQueue_(
+          std::make_shared<ReceiveQueue>(classic_->getIOExecutorProvider()->get(), kReceiveQueueCapacity)),
+      currentLayout_(std::make_shared<SegmentLayout>()) {}
+
+Future<void> QueueConsumerImpl::start() {
+    dagWatch_ = std::make_shared<DagWatchSession>(classic_, config_.topic, /*createIfMissing*/ true);
+    std::weak_ptr<QueueConsumerImpl> weak = weak_from_this();
+    dagWatch_->setLayoutChangeListener(
+        [weak](const SegmentLayout& newLayout, const SegmentLayout& oldLayout) {
+            if (auto self = weak.lock()) self->onLayoutChange(newLayout, oldLayout);
+        });
+    dagWatch_->start().addListener([weak](const Expected<SegmentLayout>& result) {
+        if (auto self = weak.lock()) self->onStartResult(result);
+    });
+    return startPromise_.getFuture();
+}
+
+void QueueConsumerImpl::onStartResult(const Expected<SegmentLayout>& result) {
+    // Only the failure path (see StProducerImpl::onStartResult): the layout listener drives the
+    // success path — subscribe the initial segments and complete startPromise_.
+    if (!result) startPromise_.setError(result.error());
+}
+
+void QueueConsumerImpl::onLayoutChange(const SegmentLayout& newLayout, const SegmentLayout& /*oldLayout*/) {
+    // Subscribe active AND sealed segments: a sealed segment may still hold undrained messages.
+    std::vector<Segment> target;
+    target.reserve(newLayout.activeSegments().size() + newLayout.sealedSegments().size());
+    for (const auto& segment : newLayout.activeSegments()) target.push_back(segment);
+    for (const auto& segment : newLayout.sealedSegments()) target.push_back(segment);
+
+    std::vector<Future<pulsar::Consumer>> retired;
+    std::vector<Segment> toAdd;
+    bool first = false;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        first = !sawFirstLayout_;
+        sawFirstLayout_ = true;
+        currentLayout_ = std::make_shared<SegmentLayout>(newLayout);
+
+        std::unordered_set<std::uint64_t> targetIds;
+        for (const auto& segment : target) targetIds.insert(segment.segmentId);
+        for (auto it = segmentConsumers_.begin(); it != segmentConsumers_.end();) {
+            if (targetIds.find(it->first) == targetIds.end()) {
+                retired.push_back(std::move(it->second));
+                it = segmentConsumers_.erase(it);
+            } else {
+                ++it;
+            }
+        }
+        // Forget segments that have left the DAG so a future segment id can never be mistaken for a
+        // previously-drained one.
+        for (auto it = drainedSegments_.begin(); it != drainedSegments_.end();) {
+            it = targetIds.count(*it) ? std::next(it) : drainedSegments_.erase(it);
+        }
+        for (const auto& segment : target) {
+            if (segmentConsumers_.find(segment.segmentId) == segmentConsumers_.end() &&
+                drainedSegments_.find(segment.segmentId) == drainedSegments_.end())
+                toAdd.push_back(segment);
+        }
+    }
+
+    for (auto& future : retired) {
+        future.addListener([](const Expected<pulsar::Consumer>& result) {
+            if (result) {
+                pulsar::Consumer consumer = *result;
+                consumer.closeAsync([](pulsar::Result) {});
+            }
+        });
+    }
+
+    if (first) {
+        if (toAdd.empty()) {
+            startPromise_.setSuccess();
+            return;
+        }
+        auto remaining = std::make_shared<std::atomic<int>>(static_cast<int>(toAdd.size()));
+        for (const auto& segment : toAdd) {
+            getOrCreateSegmentConsumerAsync(segment).addListener(
+                [self = shared_from_this(), remaining](const Expected<pulsar::Consumer>& result) {
+                    if (!result) {
+                        self->startPromise_.setError(result.error());  // first error wins (idempotent)
+                        return;
+                    }
+                    if (remaining->fetch_sub(1) == 1) self->startPromise_.setSuccess();
+                });
+        }
+    } else {
+        for (const auto& segment : toAdd) getOrCreateSegmentConsumerAsync(segment);  // best-effort
+    }
+}
+
+pulsar::ConsumerConfiguration QueueConsumerImpl::buildSegmentConfiguration(const Segment& segment) const {
+    // Build a FRESH config every time (pulsar::ConsumerConfiguration's copy ctor shares its impl).
+    pulsar::ConsumerConfiguration conf;
+    conf.setConsumerType(pulsar::ConsumerShared);
+    conf.setSchema(config_.schema);
+    conf.setSubscriptionInitialPosition(toClassicInitialPosition(config_.initialPosition));
+    if (config_.consumerName) {
+        conf.setConsumerName(*config_.consumerName + "-seg-" + std::to_string(segment.segmentId));
+    }
+    if (config_.ackPolicy.groupTime) {
+        conf.setAckGroupingTimeMs(static_cast<long>(config_.ackPolicy.groupTime->count()));
+    }
+    if (config_.ackPolicy.negativeAckRedeliveryDelay) {
+        conf.setNegativeAckRedeliveryDelayMs(
+            static_cast<long>(config_.ackPolicy.negativeAckRedeliveryDelay->count()));
+    }
+    for (const auto& [key, value] : config_.properties) conf.setProperty(key, value);
+    if (segment.isLegacy()) conf.setProperty("__pulsar.v5.managed", "true");
+    return conf;
+}
+
+Future<pulsar::Consumer> QueueConsumerImpl::getOrCreateSegmentConsumerAsync(const Segment& segment) {
+    detail::Promise<pulsar::Consumer> promise;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        if (auto it = segmentConsumers_.find(segment.segmentId); it != segmentConsumers_.end()) {
+            return it->second;
+        }
+        segmentConsumers_.insert_or_assign(segment.segmentId, promise.getFuture());
+    }
+
+    const pulsar::ConsumerConfiguration conf = buildSegmentConfiguration(segment);
+    const std::string attachTopic = segment.attachTopicName();
+    const std::uint64_t segmentId = segment.segmentId;
+    auto self = shared_from_this();
+    classic_->subscribeSegmentAsync(
+        attachTopic, config_.subscriptionName, conf,
+        [self, promise, segmentId](std::variant<pulsar::Error, pulsar::Consumer> result) {
+            if (auto* consumer = std::get_if<pulsar::Consumer>(&result)) {
+                // pulsar::Consumer is a copyable handle (its virtual dtor suppresses the move ctor),
+                // so this is a shared-impl copy, not a deep copy.
+                pulsar::Consumer c = *consumer;
+                self->startReceiveLoop(c, segmentId);
+                promise.setValue(c);
+            } else {
+                // Evict the failed subscribe so a later reconcile retries this segment.
+                {
+                    std::lock_guard<std::mutex> lock(self->mutex_);
+                    self->segmentConsumers_.erase(segmentId);
+                }
+                promise.setError(std::get<pulsar::Error>(result));
+            }
+        });
+    return promise.getFuture();
+}
+
+void QueueConsumerImpl::startReceiveLoop(pulsar::Consumer consumer, std::uint64_t segmentId) {
+    if (closed_.load()) return;
+    auto self = shared_from_this();
+    consumer.receiveAsync([self, consumer, segmentId](pulsar::Result result, const pulsar::Message& message) {
+        if (result != pulsar::ResultOk) {
+            if (result == pulsar::ResultTopicTerminated) {
+                // A sealed segment fully drained: close its consumer, drop it from the cache, and
+                // remember it drained so a later reconcile does not re-subscribe the still-in-DAG
+                // sealed segment (which would redeliver its unacked messages as duplicates).
+                {

Review Comment:
   **[BUG] Closing a drained sealed segment loses the acks for messages still in the mux queue or in the application's hands**
   
   `ResultTopicTerminated` only tells you the *classic* consumer's prefetch queue is empty. It says nothing about the messages this loop already pulled out of it: up to `kReceiveQueueCapacity` (1000) of them may still be sitting in `receiveQueue_`, plus any already handed to the application and not yet acked.
   
   Erasing `segmentConsumers_[segmentId]` here removes the only ack-routing handle for those messages, and `drainedSegments_.insert(segmentId)` guarantees the segment is never reattached for the life of this consumer. `acknowledge()` then resolves `segmentConsumerFor(id)` to an errored future at line 244, and the listener at lines 252-257 has no error branch — the ack is silently discarded.
   
   Failure scenario: a sealed segment holds 500 undrained messages. The receive loop pulls all 500 into the mux queue and the broker reports end-of-topic; this handler closes the consumer. The application then receives and acks all 500 from the mux queue. Every one of those acks is dropped. The durable cursor still holds them, so they are redelivered in full to the next consumer on this subscription, and the sealed segment never becomes eligible for removal.
   
   This is the half of your earlier thread with Copilot that `drainedSegments_` did not close — it fixed the duplicate-redelivery symptom, but the acks are still lost. A refcount of outstanding messages per segment, with the close deferred until it reaches zero, would cover both.



##########
lib/st/QueueConsumerImpl.cc:
##########
@@ -0,0 +1,314 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "QueueConsumerImpl.h"
+
+#include <cstdint>
+#include <string>
+#include <unordered_set>
+#include <utility>
+#include <variant>
+#include <vector>
+
+#include "MessageIdImpl.h"
+#include "MessageImpl.h"
+#include "lib/LogUtils.h"
+
+DECLARE_LOG_OBJECT()
+
+namespace pulsar::st {
+
+namespace {
+
+pulsar::InitialPosition toClassicInitialPosition(SubscriptionInitialPosition position) {
+    return position == SubscriptionInitialPosition::Earliest ? pulsar::InitialPositionEarliest
+                                                             : pulsar::InitialPositionLatest;
+}
+
+}  // namespace
+
+QueueConsumerImpl::QueueConsumerImpl(pulsar::ClientImplPtr classic, QueueConsumerConfig config)
+    : classic_(std::move(classic)),
+      config_(std::move(config)),
+      topic_(config_.topic),
+      subscription_(config_.subscriptionName),
+      consumerName_(config_.consumerName.value_or(std::string{})),
+      receiveQueue_(
+          std::make_shared<ReceiveQueue>(classic_->getIOExecutorProvider()->get(), kReceiveQueueCapacity)),
+      currentLayout_(std::make_shared<SegmentLayout>()) {}
+
+Future<void> QueueConsumerImpl::start() {
+    dagWatch_ = std::make_shared<DagWatchSession>(classic_, config_.topic, /*createIfMissing*/ true);
+    std::weak_ptr<QueueConsumerImpl> weak = weak_from_this();
+    dagWatch_->setLayoutChangeListener(
+        [weak](const SegmentLayout& newLayout, const SegmentLayout& oldLayout) {
+            if (auto self = weak.lock()) self->onLayoutChange(newLayout, oldLayout);
+        });
+    dagWatch_->start().addListener([weak](const Expected<SegmentLayout>& result) {
+        if (auto self = weak.lock()) self->onStartResult(result);
+    });
+    return startPromise_.getFuture();
+}
+
+void QueueConsumerImpl::onStartResult(const Expected<SegmentLayout>& result) {
+    // Only the failure path (see StProducerImpl::onStartResult): the layout listener drives the
+    // success path — subscribe the initial segments and complete startPromise_.
+    if (!result) startPromise_.setError(result.error());
+}
+
+void QueueConsumerImpl::onLayoutChange(const SegmentLayout& newLayout, const SegmentLayout& /*oldLayout*/) {
+    // Subscribe active AND sealed segments: a sealed segment may still hold undrained messages.
+    std::vector<Segment> target;
+    target.reserve(newLayout.activeSegments().size() + newLayout.sealedSegments().size());
+    for (const auto& segment : newLayout.activeSegments()) target.push_back(segment);
+    for (const auto& segment : newLayout.sealedSegments()) target.push_back(segment);
+
+    std::vector<Future<pulsar::Consumer>> retired;
+    std::vector<Segment> toAdd;
+    bool first = false;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        first = !sawFirstLayout_;
+        sawFirstLayout_ = true;
+        currentLayout_ = std::make_shared<SegmentLayout>(newLayout);
+
+        std::unordered_set<std::uint64_t> targetIds;
+        for (const auto& segment : target) targetIds.insert(segment.segmentId);
+        for (auto it = segmentConsumers_.begin(); it != segmentConsumers_.end();) {
+            if (targetIds.find(it->first) == targetIds.end()) {
+                retired.push_back(std::move(it->second));
+                it = segmentConsumers_.erase(it);
+            } else {
+                ++it;
+            }
+        }
+        // Forget segments that have left the DAG so a future segment id can never be mistaken for a
+        // previously-drained one.
+        for (auto it = drainedSegments_.begin(); it != drainedSegments_.end();) {
+            it = targetIds.count(*it) ? std::next(it) : drainedSegments_.erase(it);
+        }
+        for (const auto& segment : target) {
+            if (segmentConsumers_.find(segment.segmentId) == segmentConsumers_.end() &&
+                drainedSegments_.find(segment.segmentId) == drainedSegments_.end())
+                toAdd.push_back(segment);
+        }
+    }
+
+    for (auto& future : retired) {
+        future.addListener([](const Expected<pulsar::Consumer>& result) {
+            if (result) {
+                pulsar::Consumer consumer = *result;
+                consumer.closeAsync([](pulsar::Result) {});
+            }
+        });
+    }
+
+    if (first) {
+        if (toAdd.empty()) {
+            startPromise_.setSuccess();
+            return;
+        }
+        auto remaining = std::make_shared<std::atomic<int>>(static_cast<int>(toAdd.size()));
+        for (const auto& segment : toAdd) {
+            getOrCreateSegmentConsumerAsync(segment).addListener(
+                [self = shared_from_this(), remaining](const Expected<pulsar::Consumer>& result) {
+                    if (!result) {
+                        self->startPromise_.setError(result.error());  // first error wins (idempotent)
+                        return;
+                    }
+                    if (remaining->fetch_sub(1) == 1) self->startPromise_.setSuccess();
+                });
+        }
+    } else {
+        for (const auto& segment : toAdd) getOrCreateSegmentConsumerAsync(segment);  // best-effort
+    }
+}
+
+pulsar::ConsumerConfiguration QueueConsumerImpl::buildSegmentConfiguration(const Segment& segment) const {
+    // Build a FRESH config every time (pulsar::ConsumerConfiguration's copy ctor shares its impl).
+    pulsar::ConsumerConfiguration conf;
+    conf.setConsumerType(pulsar::ConsumerShared);
+    conf.setSchema(config_.schema);
+    conf.setSubscriptionInitialPosition(toClassicInitialPosition(config_.initialPosition));
+    if (config_.consumerName) {
+        conf.setConsumerName(*config_.consumerName + "-seg-" + std::to_string(segment.segmentId));
+    }
+    if (config_.ackPolicy.groupTime) {
+        conf.setAckGroupingTimeMs(static_cast<long>(config_.ackPolicy.groupTime->count()));
+    }
+    if (config_.ackPolicy.negativeAckRedeliveryDelay) {
+        conf.setNegativeAckRedeliveryDelayMs(
+            static_cast<long>(config_.ackPolicy.negativeAckRedeliveryDelay->count()));
+    }
+    for (const auto& [key, value] : config_.properties) conf.setProperty(key, value);
+    if (segment.isLegacy()) conf.setProperty("__pulsar.v5.managed", "true");
+    return conf;
+}
+
+Future<pulsar::Consumer> QueueConsumerImpl::getOrCreateSegmentConsumerAsync(const Segment& segment) {
+    detail::Promise<pulsar::Consumer> promise;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        if (auto it = segmentConsumers_.find(segment.segmentId); it != segmentConsumers_.end()) {
+            return it->second;
+        }
+        segmentConsumers_.insert_or_assign(segment.segmentId, promise.getFuture());
+    }
+
+    const pulsar::ConsumerConfiguration conf = buildSegmentConfiguration(segment);
+    const std::string attachTopic = segment.attachTopicName();
+    const std::uint64_t segmentId = segment.segmentId;
+    auto self = shared_from_this();
+    classic_->subscribeSegmentAsync(
+        attachTopic, config_.subscriptionName, conf,
+        [self, promise, segmentId](std::variant<pulsar::Error, pulsar::Consumer> result) {
+            if (auto* consumer = std::get_if<pulsar::Consumer>(&result)) {
+                // pulsar::Consumer is a copyable handle (its virtual dtor suppresses the move ctor),
+                // so this is a shared-impl copy, not a deep copy.
+                pulsar::Consumer c = *consumer;
+                self->startReceiveLoop(c, segmentId);
+                promise.setValue(c);
+            } else {
+                // Evict the failed subscribe so a later reconcile retries this segment.
+                {
+                    std::lock_guard<std::mutex> lock(self->mutex_);
+                    self->segmentConsumers_.erase(segmentId);
+                }
+                promise.setError(std::get<pulsar::Error>(result));
+            }
+        });
+    return promise.getFuture();
+}
+
+void QueueConsumerImpl::startReceiveLoop(pulsar::Consumer consumer, std::uint64_t segmentId) {
+    if (closed_.load()) return;
+    auto self = shared_from_this();
+    consumer.receiveAsync([self, consumer, segmentId](pulsar::Result result, const pulsar::Message& message) {
+        if (result != pulsar::ResultOk) {
+            if (result == pulsar::ResultTopicTerminated) {
+                // A sealed segment fully drained: close its consumer, drop it from the cache, and
+                // remember it drained so a later reconcile does not re-subscribe the still-in-DAG
+                // sealed segment (which would redeliver its unacked messages as duplicates).
+                {
+                    std::lock_guard<std::mutex> lock(self->mutex_);
+                    self->segmentConsumers_.erase(segmentId);
+                    self->drainedSegments_.insert(segmentId);
+                }
+                pulsar::Consumer done = consumer;
+                done.closeAsync([](pulsar::Result) {});
+            }
+            // Otherwise (AlreadyClosed / consumer closing) just stop the loop.
+            return;
+        }
+        MessageId id = MessageIdFactory::create(message.getMessageId(), static_cast<std::int64_t>(segmentId));
+        auto messageImpl = std::make_shared<MessageImpl>(message, std::move(id));
+        // Re-arm only once the fan-in queue has room, so a slow consumer throttles this segment.
+        self->receiveQueue_->offer(std::move(messageImpl))

Review Comment:
   **[BUG] The segment receive loop recurses synchronously per message and can exhaust the stack**
   
   This re-arm is not a loop — it is recursion, and nothing guarantees it unwinds:
   
   - `Consumer::receiveAsync` (`lib/Consumer.cc:82`) is a thin forwarder, and `ConsumerImpl::receiveAsync` (`lib/ConsumerImpl.cc:1201-1207`) invokes `callback(ResultOk, msg)` **inline** when `incomingMessages_.pop()` succeeds.
   - `offer()` returns an **already-completed** future whenever the queue has room (`ReceiveQueue.cc:134-137`).
   - `SharedState::addListener` (`include/pulsar/st/detail/SharedState.h:58-67`) runs the listener **synchronously** when the result is already available.
   
   So the listener calls `startReceiveLoop` on the same stack, which calls `receiveAsync`, which calls back inline, and so on — roughly 8 frames per message.
   
   The counter-intuitive part: an application that is actively receiving makes this *worse*. When a receive is already parked, `offer()` hands the message straight to the receiver (`ReceiveQueue.cc:118-122`) without touching `buffer_`, so the buffer never reaches capacity and the back-pressure brake never engages. The chain then breaks only when the classic prefetch queue runs dry — and because each dequeue replenishes flow permits, broker I/O refills it concurrently from another thread while this one is still recursing.
   
   Failure scenario: a consumer attaches to a segment with a large backlog and drains it promptly. The default `receiverQueueSize` is 1000 (nothing here overrides it), so even without concurrent replenishment the chain reaches ~1000 nested iterations — several thousand frames — on a listener-executor thread whose stack is well under a megabyte. Under sustained throughput there is no bound at all.
   
   Posting the re-arm to the executor instead of continuing inline would make it a loop again.



##########
lib/st/QueueConsumerImpl.cc:
##########
@@ -0,0 +1,314 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "QueueConsumerImpl.h"
+
+#include <cstdint>
+#include <string>
+#include <unordered_set>
+#include <utility>
+#include <variant>
+#include <vector>
+
+#include "MessageIdImpl.h"
+#include "MessageImpl.h"
+#include "lib/LogUtils.h"
+
+DECLARE_LOG_OBJECT()
+
+namespace pulsar::st {
+
+namespace {
+
+pulsar::InitialPosition toClassicInitialPosition(SubscriptionInitialPosition position) {
+    return position == SubscriptionInitialPosition::Earliest ? pulsar::InitialPositionEarliest
+                                                             : pulsar::InitialPositionLatest;
+}
+
+}  // namespace
+
+QueueConsumerImpl::QueueConsumerImpl(pulsar::ClientImplPtr classic, QueueConsumerConfig config)
+    : classic_(std::move(classic)),
+      config_(std::move(config)),
+      topic_(config_.topic),
+      subscription_(config_.subscriptionName),
+      consumerName_(config_.consumerName.value_or(std::string{})),
+      receiveQueue_(
+          std::make_shared<ReceiveQueue>(classic_->getIOExecutorProvider()->get(), kReceiveQueueCapacity)),
+      currentLayout_(std::make_shared<SegmentLayout>()) {}
+
+Future<void> QueueConsumerImpl::start() {
+    dagWatch_ = std::make_shared<DagWatchSession>(classic_, config_.topic, /*createIfMissing*/ true);
+    std::weak_ptr<QueueConsumerImpl> weak = weak_from_this();
+    dagWatch_->setLayoutChangeListener(
+        [weak](const SegmentLayout& newLayout, const SegmentLayout& oldLayout) {
+            if (auto self = weak.lock()) self->onLayoutChange(newLayout, oldLayout);
+        });
+    dagWatch_->start().addListener([weak](const Expected<SegmentLayout>& result) {
+        if (auto self = weak.lock()) self->onStartResult(result);
+    });
+    return startPromise_.getFuture();
+}
+
+void QueueConsumerImpl::onStartResult(const Expected<SegmentLayout>& result) {
+    // Only the failure path (see StProducerImpl::onStartResult): the layout listener drives the
+    // success path — subscribe the initial segments and complete startPromise_.
+    if (!result) startPromise_.setError(result.error());
+}
+
+void QueueConsumerImpl::onLayoutChange(const SegmentLayout& newLayout, const SegmentLayout& /*oldLayout*/) {
+    // Subscribe active AND sealed segments: a sealed segment may still hold undrained messages.
+    std::vector<Segment> target;
+    target.reserve(newLayout.activeSegments().size() + newLayout.sealedSegments().size());
+    for (const auto& segment : newLayout.activeSegments()) target.push_back(segment);
+    for (const auto& segment : newLayout.sealedSegments()) target.push_back(segment);
+
+    std::vector<Future<pulsar::Consumer>> retired;
+    std::vector<Segment> toAdd;
+    bool first = false;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        first = !sawFirstLayout_;
+        sawFirstLayout_ = true;
+        currentLayout_ = std::make_shared<SegmentLayout>(newLayout);
+
+        std::unordered_set<std::uint64_t> targetIds;
+        for (const auto& segment : target) targetIds.insert(segment.segmentId);
+        for (auto it = segmentConsumers_.begin(); it != segmentConsumers_.end();) {
+            if (targetIds.find(it->first) == targetIds.end()) {
+                retired.push_back(std::move(it->second));
+                it = segmentConsumers_.erase(it);
+            } else {
+                ++it;
+            }
+        }
+        // Forget segments that have left the DAG so a future segment id can never be mistaken for a
+        // previously-drained one.
+        for (auto it = drainedSegments_.begin(); it != drainedSegments_.end();) {
+            it = targetIds.count(*it) ? std::next(it) : drainedSegments_.erase(it);
+        }
+        for (const auto& segment : target) {
+            if (segmentConsumers_.find(segment.segmentId) == segmentConsumers_.end() &&
+                drainedSegments_.find(segment.segmentId) == drainedSegments_.end())
+                toAdd.push_back(segment);
+        }
+    }
+
+    for (auto& future : retired) {
+        future.addListener([](const Expected<pulsar::Consumer>& result) {
+            if (result) {
+                pulsar::Consumer consumer = *result;
+                consumer.closeAsync([](pulsar::Result) {});
+            }
+        });
+    }
+
+    if (first) {
+        if (toAdd.empty()) {
+            startPromise_.setSuccess();
+            return;
+        }
+        auto remaining = std::make_shared<std::atomic<int>>(static_cast<int>(toAdd.size()));
+        for (const auto& segment : toAdd) {
+            getOrCreateSegmentConsumerAsync(segment).addListener(
+                [self = shared_from_this(), remaining](const Expected<pulsar::Consumer>& result) {
+                    if (!result) {
+                        self->startPromise_.setError(result.error());  // first error wins (idempotent)
+                        return;
+                    }
+                    if (remaining->fetch_sub(1) == 1) self->startPromise_.setSuccess();
+                });
+        }
+    } else {
+        for (const auto& segment : toAdd) getOrCreateSegmentConsumerAsync(segment);  // best-effort
+    }
+}
+
+pulsar::ConsumerConfiguration QueueConsumerImpl::buildSegmentConfiguration(const Segment& segment) const {
+    // Build a FRESH config every time (pulsar::ConsumerConfiguration's copy ctor shares its impl).
+    pulsar::ConsumerConfiguration conf;
+    conf.setConsumerType(pulsar::ConsumerShared);
+    conf.setSchema(config_.schema);
+    conf.setSubscriptionInitialPosition(toClassicInitialPosition(config_.initialPosition));
+    if (config_.consumerName) {
+        conf.setConsumerName(*config_.consumerName + "-seg-" + std::to_string(segment.segmentId));
+    }
+    if (config_.ackPolicy.groupTime) {
+        conf.setAckGroupingTimeMs(static_cast<long>(config_.ackPolicy.groupTime->count()));
+    }
+    if (config_.ackPolicy.negativeAckRedeliveryDelay) {
+        conf.setNegativeAckRedeliveryDelayMs(
+            static_cast<long>(config_.ackPolicy.negativeAckRedeliveryDelay->count()));
+    }
+    for (const auto& [key, value] : config_.properties) conf.setProperty(key, value);
+    if (segment.isLegacy()) conf.setProperty("__pulsar.v5.managed", "true");
+    return conf;
+}
+
+Future<pulsar::Consumer> QueueConsumerImpl::getOrCreateSegmentConsumerAsync(const Segment& segment) {
+    detail::Promise<pulsar::Consumer> promise;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        if (auto it = segmentConsumers_.find(segment.segmentId); it != segmentConsumers_.end()) {
+            return it->second;
+        }
+        segmentConsumers_.insert_or_assign(segment.segmentId, promise.getFuture());
+    }
+
+    const pulsar::ConsumerConfiguration conf = buildSegmentConfiguration(segment);
+    const std::string attachTopic = segment.attachTopicName();
+    const std::uint64_t segmentId = segment.segmentId;
+    auto self = shared_from_this();
+    classic_->subscribeSegmentAsync(
+        attachTopic, config_.subscriptionName, conf,
+        [self, promise, segmentId](std::variant<pulsar::Error, pulsar::Consumer> result) {
+            if (auto* consumer = std::get_if<pulsar::Consumer>(&result)) {
+                // pulsar::Consumer is a copyable handle (its virtual dtor suppresses the move ctor),
+                // so this is a shared-impl copy, not a deep copy.
+                pulsar::Consumer c = *consumer;
+                self->startReceiveLoop(c, segmentId);
+                promise.setValue(c);
+            } else {
+                // Evict the failed subscribe so a later reconcile retries this segment.
+                {
+                    std::lock_guard<std::mutex> lock(self->mutex_);
+                    self->segmentConsumers_.erase(segmentId);
+                }
+                promise.setError(std::get<pulsar::Error>(result));
+            }
+        });
+    return promise.getFuture();
+}
+
+void QueueConsumerImpl::startReceiveLoop(pulsar::Consumer consumer, std::uint64_t segmentId) {
+    if (closed_.load()) return;
+    auto self = shared_from_this();
+    consumer.receiveAsync([self, consumer, segmentId](pulsar::Result result, const pulsar::Message& message) {
+        if (result != pulsar::ResultOk) {
+            if (result == pulsar::ResultTopicTerminated) {
+                // A sealed segment fully drained: close its consumer, drop it from the cache, and
+                // remember it drained so a later reconcile does not re-subscribe the still-in-DAG
+                // sealed segment (which would redeliver its unacked messages as duplicates).
+                {
+                    std::lock_guard<std::mutex> lock(self->mutex_);
+                    self->segmentConsumers_.erase(segmentId);
+                    self->drainedSegments_.insert(segmentId);
+                }
+                pulsar::Consumer done = consumer;
+                done.closeAsync([](pulsar::Result) {});
+            }
+            // Otherwise (AlreadyClosed / consumer closing) just stop the loop.
+            return;
+        }
+        MessageId id = MessageIdFactory::create(message.getMessageId(), static_cast<std::int64_t>(segmentId));
+        auto messageImpl = std::make_shared<MessageImpl>(message, std::move(id));

Review Comment:
   **[INTENT MISMATCH] Message::topic() reports the internal segment:// backing topic instead of the scalable topic**
   
   `MessageImpl` is constructed without `topicOverride`, so `topic()` falls through to `classic_.getTopicName()` — the `segment://...` backing topic.
   
   The public contract (`include/pulsar/st/Message.h:135-139`) promises "the resolved canonical topic the message was received from" and "the fully-qualified topic name". Segment topics are internal enough that the public subscribe path rejects them outright, so surfacing one back to the application inverts that boundary — and the value shifts under the application whenever a split reassigns segments.
   
   `config_.topic` is right there in the impl; passing it as the override would keep the abstraction intact.



##########
lib/st/ReceiveQueue.cc:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "ReceiveQueue.h"
+
+#include <utility>
+
+#include "MessageImpl.h"
+#include "lib/ExecutorService.h"
+
+namespace pulsar::st {
+
+ReceiveQueue::ReceiveQueue(pulsar::ExecutorServicePtr executor, std::size_t capacity)
+    : executor_(std::move(executor)), capacity_(capacity) {}
+
+std::deque<detail::Promise<void>> ReceiveQueue::takeCapacityWaitersIfRoomLocked() {
+    std::deque<detail::Promise<void>> toSignal;
+    if (buffer_.size() < capacity_ && !capacityWaiters_.empty()) {
+        toSignal = std::move(capacityWaiters_);
+        capacityWaiters_.clear();
+    }
+    return toSignal;
+}
+
+Future<MessageImplPtr> ReceiveQueue::receiveAsync() {
+    detail::Promise<MessageImplPtr> promise;
+    MessageImplPtr message;
+    std::deque<detail::Promise<void>> toSignal;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        if (closed_) {
+            promise.setError(Error{ResultAlreadyClosed, "consumer is closed"});
+            return promise.getFuture();
+        }
+        if (!buffer_.empty()) {
+            message = std::move(buffer_.front());
+            buffer_.pop_front();
+            toSignal = takeCapacityWaitersIfRoomLocked();
+        } else {
+            pendingReceives_.emplace(nextReceiveId_++, promise);
+        }
+    }
+    for (auto& waiter : toSignal) waiter.setSuccess();
+    if (message) promise.setValue(std::move(message));
+    return promise.getFuture();
+}
+
+Future<MessageImplPtr> ReceiveQueue::receiveAsync(std::chrono::milliseconds timeout) {
+    detail::Promise<MessageImplPtr> promise;
+    MessageImplPtr message;
+    std::deque<detail::Promise<void>> toSignal;
+    std::uint64_t receiveId = 0;
+    bool parked = false;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        if (closed_) {
+            promise.setError(Error{ResultAlreadyClosed, "consumer is closed"});
+            return promise.getFuture();
+        }
+        if (!buffer_.empty()) {
+            message = std::move(buffer_.front());
+            buffer_.pop_front();
+            toSignal = takeCapacityWaitersIfRoomLocked();
+        } else {
+            receiveId = nextReceiveId_++;
+            pendingReceives_.emplace(receiveId, promise);
+            parked = true;
+        }
+    }
+    for (auto& waiter : toSignal) waiter.setSuccess();
+    if (message) {
+        promise.setValue(std::move(message));
+        return promise.getFuture();
+    }
+    if (parked) {
+        auto timer = executor_->createDeadlineTimer();

Review Comment:
   **[QUALITY] A timed receive never cancels its timer, holding the queue and promise alive until the deadline**
   
   When a message arrives first, the timer is left to fire and no-op — but until it does it keeps `self` (the whole `ReceiveQueue`), the promise and the timer object alive.
   
   Under a steady stream of `receiveAsync(timeout)` calls with a generous timeout, live timers accumulate in proportion to receive rate × timeout rather than to the number of outstanding receives.
   
   Copilot raised this earlier and the thread was resolved, but the follow-up commit touched only `QueueConsumerImpl.{cc,h}`, so the code is unchanged. Flagging because I think the point stands — cancelling the timer on the delivery path would settle it.



##########
tests/st/StQueueConsumerE2ETest.cc:
##########
@@ -0,0 +1,203 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// End-to-end queue-consumer tests against a real scalable-topics broker: a produce -> consume
+// round-trip over a Shared subscription. Gated on the PULSAR_ST_E2E environment variable so the
+// ordinary (broker-free) unit-test run skips them; the docker harness sets it. The broker URL
+// defaults to the standard test service and can be overridden with PULSAR_ST_E2E_SERVICE_URL.
+#include <gtest/gtest.h>
+#include <pulsar/st/Client.h>
+
+#include <chrono>
+#include <cstdint>
+#include <cstdlib>
+#include <ctime>
+#include <set>
+#include <string>
+#include <thread>
+#include <vector>
+
+#include "lib/st/MessageIdImpl.h"
+#include "tests/HttpHelper.h"
+
+using namespace pulsar::st;
+
+namespace {
+
+bool e2eEnabled() { return std::getenv("PULSAR_ST_E2E") != nullptr; }
+
+std::string serviceUrl() {
+    const char* url = std::getenv("PULSAR_ST_E2E_SERVICE_URL");
+    return url != nullptr ? url : "pulsar://localhost:6650";
+}
+
+std::string adminUrl() {
+    const char* url = std::getenv("PULSAR_ST_E2E_ADMIN_URL");
+    return url != nullptr ? url : "http://localhost:8080";
+}
+
+// A fresh topic name per test run so tests never collide with one another or with a topic left on a
+// reused broker (the same convention the classic tests use).
+std::string uniqueName(const std::string& prefix) {
+    static int counter = 0;
+    return prefix + "-" + std::to_string(std::time(nullptr)) + "-" + std::to_string(counter++);
+}
+
+std::string topicUrl(const std::string& name) { return "topic://public/default/" + name; }
+
+// The admin REST base for a scalable topic under public/default.
+std::string scalablePath(const std::string& name) {
+    return adminUrl() + "/admin/v2/scalable/public/default/" + name;
+}
+
+// Create a scalable topic with the given number of initial segments. Retries while the
+// scalable-topics controller finishes coming up after broker start (only the first test waits).
+bool createScalableTopic(const std::string& name, int numInitialSegments = 1) {
+    const std::string url = scalablePath(name) + "?numInitialSegments=" + std::to_string(numInitialSegments);
+    for (int attempt = 0; attempt < 30; attempt++) {
+        const int code = makePutRequest(url, "");
+        if (code >= 200 && code < 300) return true;
+        std::this_thread::sleep_for(std::chrono::seconds(1));
+    }
+    return false;
+}
+
+// Split a segment into two half-range children (POST .../split/{segmentId}).
+bool splitSegment(const std::string& name, std::int64_t segmentId) {
+    const int code = makePostRequest(scalablePath(name) + "/split/" + std::to_string(segmentId), "");
+    return code >= 200 && code < 300;
+}
+
+// The segment id carried by a received message id — the fan-in stamps it on every message.
+std::int64_t segmentIdOf(const MessageId& id) {
+    const auto& impl = MessageIdFactory::impl(id);
+    return impl ? impl->segmentId : MessageIdImpl::kNoSegment;
+}
+
+// Give each message plenty of time to arrive; a healthy broker delivers in milliseconds.
+constexpr std::chrono::seconds kReceiveTimeout{20};
+
+TEST(StQueueConsumerE2ETest, testProduceThenConsumeRoundTrip) {
+    if (!e2eEnabled()) GTEST_SKIP() << "set PULSAR_ST_E2E=1 to run against a scalable-topics broker";
+
+    const std::string name = uniqueName("st-e2e-queue");
+    ASSERT_TRUE(createScalableTopic(name)) << "failed to create scalable topic " << name;
+    const std::string topic = topicUrl(name);
+
+    auto clientResult = PulsarClient::builder().serviceUrl(serviceUrl()).build();
+    ASSERT_TRUE(clientResult) << clientResult.error();
+    PulsarClient client = std::move(clientResult).value();
+
+    // Subscribe first (Earliest) so the subscription and its per-segment cursors exist before we
+    // publish — every produced message is then guaranteed to be delivered.
+    auto consumerResult = client.newQueueConsumer(Schema<std::string>{})
+                              .topic(topic)
+                              .subscriptionName("sub")
+                              .subscriptionInitialPosition(SubscriptionInitialPosition::Earliest)
+                              .subscribe();
+    ASSERT_TRUE(consumerResult) << consumerResult.error();
+    QueueConsumer<std::string> consumer = std::move(consumerResult).value();
+
+    auto producerResult = client.newProducer(Schema<std::string>{}).topic(topic).create();
+    ASSERT_TRUE(producerResult) << producerResult.error();
+    Producer<std::string> producer = std::move(producerResult).value();
+
+    constexpr int kCount = 25;
+    std::set<std::string> produced;
+    for (int i = 0; i < kCount; i++) {
+        std::string value = "v-" + std::to_string(i);
+        auto sent = producer.newMessage().key("key-" + std::to_string(i % 4)).value(value).send();
+        ASSERT_TRUE(sent) << "send " << i << " failed: " << sent.error();
+        produced.insert(value);
+    }
+    ASSERT_TRUE(producer.flush());
+
+    // Receive exactly kCount messages; a Shared subscription gives no cross-segment order, so
+    // compare the received payloads as a set rather than a sequence.
+    std::set<std::string> received;
+    for (int i = 0; i < kCount; i++) {
+        auto message = consumer.receive(kReceiveTimeout);
+        ASSERT_TRUE(message) << "receive " << i << " failed: " << message.error();
+        EXPECT_GE(segmentIdOf(message->id()), 0) << "received message " << i << " has no real segment id";
+        received.insert(message->value());
+        consumer.acknowledge(message->id());
+    }
+    EXPECT_EQ(received, produced) << "consumed payloads did not match what was produced";
+
+    EXPECT_TRUE(producer.close());
+    EXPECT_TRUE(consumer.close());
+    EXPECT_TRUE(client.close());
+}
+
+// Consume from a topic split (via REST) into two active segments and assert the messages actually
+// arrive from both — this is the fan-in the queue consumer exists for: one Shared subscription
+// multiplexed across a per-segment classic consumer each, drained through the mux receive queue.
+// The single-segment round-trip above never exercises multi-segment fan-in.
+TEST(StQueueConsumerE2ETest, testConsumeAcrossSplitSegments) {
+    if (!e2eEnabled()) GTEST_SKIP() << "set PULSAR_ST_E2E=1 to run against a scalable-topics broker";
+
+    const std::string name = uniqueName("st-e2e-queue-split");
+    ASSERT_TRUE(createScalableTopic(name)) << "failed to create scalable topic " << name;
+    ASSERT_TRUE(splitSegment(name, 0)) << "failed to split segment 0 of " << name;

Review Comment:
   **[QUALITY] Neither e2e test drains a sealed segment — the PR's headline scenario is untested**
   
   The split happens here, immediately after `createScalableTopic` and before the producer exists. Segment 0 is therefore sealed while empty, and every message goes to the two active children.
   
   So this test covers cross-segment fan-in, but never the sealed-segment drain — no `ResultTopicTerminated` is ever delivered, and the `drainedSegments_` logic, the close, and the ack routing for a sealed segment all go unexercised. That is the case the PR description leads with: "a split seals the parent without migrating its backlog, so its messages are only drainable through the sealed segment."
   
   A test that produces first, then splits, then consumes — asserting every pre-split message still arrives and that its acks stick (e.g. by reattaching a second consumer on the same subscription and getting nothing) — would cover the drain path, and I suspect would surface the ack-loss issue above directly.



##########
lib/st/QueueConsumerImpl.cc:
##########
@@ -0,0 +1,314 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "QueueConsumerImpl.h"
+
+#include <cstdint>
+#include <string>
+#include <unordered_set>
+#include <utility>
+#include <variant>
+#include <vector>
+
+#include "MessageIdImpl.h"
+#include "MessageImpl.h"
+#include "lib/LogUtils.h"
+
+DECLARE_LOG_OBJECT()
+
+namespace pulsar::st {
+
+namespace {
+
+pulsar::InitialPosition toClassicInitialPosition(SubscriptionInitialPosition position) {
+    return position == SubscriptionInitialPosition::Earliest ? pulsar::InitialPositionEarliest
+                                                             : pulsar::InitialPositionLatest;
+}
+
+}  // namespace
+
+QueueConsumerImpl::QueueConsumerImpl(pulsar::ClientImplPtr classic, QueueConsumerConfig config)
+    : classic_(std::move(classic)),
+      config_(std::move(config)),
+      topic_(config_.topic),
+      subscription_(config_.subscriptionName),
+      consumerName_(config_.consumerName.value_or(std::string{})),
+      receiveQueue_(
+          std::make_shared<ReceiveQueue>(classic_->getIOExecutorProvider()->get(), kReceiveQueueCapacity)),
+      currentLayout_(std::make_shared<SegmentLayout>()) {}
+
+Future<void> QueueConsumerImpl::start() {
+    dagWatch_ = std::make_shared<DagWatchSession>(classic_, config_.topic, /*createIfMissing*/ true);
+    std::weak_ptr<QueueConsumerImpl> weak = weak_from_this();
+    dagWatch_->setLayoutChangeListener(
+        [weak](const SegmentLayout& newLayout, const SegmentLayout& oldLayout) {
+            if (auto self = weak.lock()) self->onLayoutChange(newLayout, oldLayout);
+        });
+    dagWatch_->start().addListener([weak](const Expected<SegmentLayout>& result) {
+        if (auto self = weak.lock()) self->onStartResult(result);
+    });
+    return startPromise_.getFuture();
+}
+
+void QueueConsumerImpl::onStartResult(const Expected<SegmentLayout>& result) {
+    // Only the failure path (see StProducerImpl::onStartResult): the layout listener drives the
+    // success path — subscribe the initial segments and complete startPromise_.
+    if (!result) startPromise_.setError(result.error());
+}
+
+void QueueConsumerImpl::onLayoutChange(const SegmentLayout& newLayout, const SegmentLayout& /*oldLayout*/) {
+    // Subscribe active AND sealed segments: a sealed segment may still hold undrained messages.
+    std::vector<Segment> target;
+    target.reserve(newLayout.activeSegments().size() + newLayout.sealedSegments().size());
+    for (const auto& segment : newLayout.activeSegments()) target.push_back(segment);
+    for (const auto& segment : newLayout.sealedSegments()) target.push_back(segment);
+
+    std::vector<Future<pulsar::Consumer>> retired;
+    std::vector<Segment> toAdd;
+    bool first = false;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        first = !sawFirstLayout_;
+        sawFirstLayout_ = true;
+        currentLayout_ = std::make_shared<SegmentLayout>(newLayout);
+
+        std::unordered_set<std::uint64_t> targetIds;
+        for (const auto& segment : target) targetIds.insert(segment.segmentId);
+        for (auto it = segmentConsumers_.begin(); it != segmentConsumers_.end();) {
+            if (targetIds.find(it->first) == targetIds.end()) {
+                retired.push_back(std::move(it->second));
+                it = segmentConsumers_.erase(it);
+            } else {
+                ++it;
+            }
+        }
+        // Forget segments that have left the DAG so a future segment id can never be mistaken for a
+        // previously-drained one.
+        for (auto it = drainedSegments_.begin(); it != drainedSegments_.end();) {
+            it = targetIds.count(*it) ? std::next(it) : drainedSegments_.erase(it);
+        }
+        for (const auto& segment : target) {
+            if (segmentConsumers_.find(segment.segmentId) == segmentConsumers_.end() &&
+                drainedSegments_.find(segment.segmentId) == drainedSegments_.end())
+                toAdd.push_back(segment);
+        }
+    }
+
+    for (auto& future : retired) {
+        future.addListener([](const Expected<pulsar::Consumer>& result) {
+            if (result) {
+                pulsar::Consumer consumer = *result;
+                consumer.closeAsync([](pulsar::Result) {});
+            }
+        });
+    }
+
+    if (first) {
+        if (toAdd.empty()) {
+            startPromise_.setSuccess();
+            return;
+        }
+        auto remaining = std::make_shared<std::atomic<int>>(static_cast<int>(toAdd.size()));
+        for (const auto& segment : toAdd) {
+            getOrCreateSegmentConsumerAsync(segment).addListener(
+                [self = shared_from_this(), remaining](const Expected<pulsar::Consumer>& result) {
+                    if (!result) {
+                        self->startPromise_.setError(result.error());  // first error wins (idempotent)
+                        return;
+                    }
+                    if (remaining->fetch_sub(1) == 1) self->startPromise_.setSuccess();
+                });
+        }
+    } else {
+        for (const auto& segment : toAdd) getOrCreateSegmentConsumerAsync(segment);  // best-effort
+    }
+}
+
+pulsar::ConsumerConfiguration QueueConsumerImpl::buildSegmentConfiguration(const Segment& segment) const {

Review Comment:
   **[QUALITY] A configured dead-letter policy is silently ignored**
   
   `QueueConsumerBuilder::deadLetterPolicy()` (`include/pulsar/st/QueueConsumer.h:335`) stores the policy into `QueueConsumerConfig`, and nothing in `lib/st` ever reads it — this builder never calls `setDeadLetterPolicy()` on the per-segment configuration.
   
   I realise dead-lettering is explicitly deferred to a follow-up, and that is a reasonable scope call. The problem is only that the builder documents it as working ("Route messages to a dead-letter topic after repeated redelivery (spec §7.2). QueueConsumer only.") with no caveat, so an application that sets it gets a silent no-op rather than an error.
   
   Failing the subscribe when the policy is set, or at minimum logging a warning the way `acknowledge(id, txn)` already does at line 260, would keep the deferral honest until the follow-up lands.



##########
lib/st/QueueConsumerImpl.cc:
##########
@@ -0,0 +1,314 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "QueueConsumerImpl.h"
+
+#include <cstdint>
+#include <string>
+#include <unordered_set>
+#include <utility>
+#include <variant>
+#include <vector>
+
+#include "MessageIdImpl.h"
+#include "MessageImpl.h"
+#include "lib/LogUtils.h"
+
+DECLARE_LOG_OBJECT()
+
+namespace pulsar::st {
+
+namespace {
+
+pulsar::InitialPosition toClassicInitialPosition(SubscriptionInitialPosition position) {
+    return position == SubscriptionInitialPosition::Earliest ? pulsar::InitialPositionEarliest
+                                                             : pulsar::InitialPositionLatest;
+}
+
+}  // namespace
+
+QueueConsumerImpl::QueueConsumerImpl(pulsar::ClientImplPtr classic, QueueConsumerConfig config)
+    : classic_(std::move(classic)),
+      config_(std::move(config)),
+      topic_(config_.topic),
+      subscription_(config_.subscriptionName),
+      consumerName_(config_.consumerName.value_or(std::string{})),
+      receiveQueue_(
+          std::make_shared<ReceiveQueue>(classic_->getIOExecutorProvider()->get(), kReceiveQueueCapacity)),
+      currentLayout_(std::make_shared<SegmentLayout>()) {}
+
+Future<void> QueueConsumerImpl::start() {
+    dagWatch_ = std::make_shared<DagWatchSession>(classic_, config_.topic, /*createIfMissing*/ true);
+    std::weak_ptr<QueueConsumerImpl> weak = weak_from_this();
+    dagWatch_->setLayoutChangeListener(
+        [weak](const SegmentLayout& newLayout, const SegmentLayout& oldLayout) {
+            if (auto self = weak.lock()) self->onLayoutChange(newLayout, oldLayout);
+        });
+    dagWatch_->start().addListener([weak](const Expected<SegmentLayout>& result) {
+        if (auto self = weak.lock()) self->onStartResult(result);
+    });
+    return startPromise_.getFuture();
+}
+
+void QueueConsumerImpl::onStartResult(const Expected<SegmentLayout>& result) {
+    // Only the failure path (see StProducerImpl::onStartResult): the layout listener drives the
+    // success path — subscribe the initial segments and complete startPromise_.
+    if (!result) startPromise_.setError(result.error());
+}
+
+void QueueConsumerImpl::onLayoutChange(const SegmentLayout& newLayout, const SegmentLayout& /*oldLayout*/) {
+    // Subscribe active AND sealed segments: a sealed segment may still hold undrained messages.
+    std::vector<Segment> target;
+    target.reserve(newLayout.activeSegments().size() + newLayout.sealedSegments().size());
+    for (const auto& segment : newLayout.activeSegments()) target.push_back(segment);
+    for (const auto& segment : newLayout.sealedSegments()) target.push_back(segment);
+
+    std::vector<Future<pulsar::Consumer>> retired;
+    std::vector<Segment> toAdd;
+    bool first = false;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        first = !sawFirstLayout_;
+        sawFirstLayout_ = true;
+        currentLayout_ = std::make_shared<SegmentLayout>(newLayout);
+
+        std::unordered_set<std::uint64_t> targetIds;
+        for (const auto& segment : target) targetIds.insert(segment.segmentId);
+        for (auto it = segmentConsumers_.begin(); it != segmentConsumers_.end();) {
+            if (targetIds.find(it->first) == targetIds.end()) {
+                retired.push_back(std::move(it->second));
+                it = segmentConsumers_.erase(it);
+            } else {
+                ++it;
+            }
+        }
+        // Forget segments that have left the DAG so a future segment id can never be mistaken for a
+        // previously-drained one.
+        for (auto it = drainedSegments_.begin(); it != drainedSegments_.end();) {
+            it = targetIds.count(*it) ? std::next(it) : drainedSegments_.erase(it);
+        }
+        for (const auto& segment : target) {
+            if (segmentConsumers_.find(segment.segmentId) == segmentConsumers_.end() &&
+                drainedSegments_.find(segment.segmentId) == drainedSegments_.end())
+                toAdd.push_back(segment);
+        }
+    }
+
+    for (auto& future : retired) {
+        future.addListener([](const Expected<pulsar::Consumer>& result) {
+            if (result) {
+                pulsar::Consumer consumer = *result;
+                consumer.closeAsync([](pulsar::Result) {});
+            }
+        });
+    }
+
+    if (first) {
+        if (toAdd.empty()) {
+            startPromise_.setSuccess();
+            return;
+        }
+        auto remaining = std::make_shared<std::atomic<int>>(static_cast<int>(toAdd.size()));
+        for (const auto& segment : toAdd) {
+            getOrCreateSegmentConsumerAsync(segment).addListener(
+                [self = shared_from_this(), remaining](const Expected<pulsar::Consumer>& result) {
+                    if (!result) {
+                        self->startPromise_.setError(result.error());  // first error wins (idempotent)
+                        return;
+                    }
+                    if (remaining->fetch_sub(1) == 1) self->startPromise_.setSuccess();
+                });
+        }
+    } else {
+        for (const auto& segment : toAdd) getOrCreateSegmentConsumerAsync(segment);  // best-effort
+    }
+}
+
+pulsar::ConsumerConfiguration QueueConsumerImpl::buildSegmentConfiguration(const Segment& segment) const {
+    // Build a FRESH config every time (pulsar::ConsumerConfiguration's copy ctor shares its impl).
+    pulsar::ConsumerConfiguration conf;
+    conf.setConsumerType(pulsar::ConsumerShared);
+    conf.setSchema(config_.schema);
+    conf.setSubscriptionInitialPosition(toClassicInitialPosition(config_.initialPosition));
+    if (config_.consumerName) {
+        conf.setConsumerName(*config_.consumerName + "-seg-" + std::to_string(segment.segmentId));
+    }
+    if (config_.ackPolicy.groupTime) {
+        conf.setAckGroupingTimeMs(static_cast<long>(config_.ackPolicy.groupTime->count()));
+    }
+    if (config_.ackPolicy.negativeAckRedeliveryDelay) {
+        conf.setNegativeAckRedeliveryDelayMs(
+            static_cast<long>(config_.ackPolicy.negativeAckRedeliveryDelay->count()));
+    }
+    for (const auto& [key, value] : config_.properties) conf.setProperty(key, value);
+    if (segment.isLegacy()) conf.setProperty("__pulsar.v5.managed", "true");
+    return conf;
+}
+
+Future<pulsar::Consumer> QueueConsumerImpl::getOrCreateSegmentConsumerAsync(const Segment& segment) {
+    detail::Promise<pulsar::Consumer> promise;
+    {
+        std::lock_guard<std::mutex> lock(mutex_);
+        if (auto it = segmentConsumers_.find(segment.segmentId); it != segmentConsumers_.end()) {
+            return it->second;
+        }
+        segmentConsumers_.insert_or_assign(segment.segmentId, promise.getFuture());
+    }
+
+    const pulsar::ConsumerConfiguration conf = buildSegmentConfiguration(segment);
+    const std::string attachTopic = segment.attachTopicName();
+    const std::uint64_t segmentId = segment.segmentId;
+    auto self = shared_from_this();
+    classic_->subscribeSegmentAsync(
+        attachTopic, config_.subscriptionName, conf,
+        [self, promise, segmentId](std::variant<pulsar::Error, pulsar::Consumer> result) {
+            if (auto* consumer = std::get_if<pulsar::Consumer>(&result)) {
+                // pulsar::Consumer is a copyable handle (its virtual dtor suppresses the move ctor),
+                // so this is a shared-impl copy, not a deep copy.
+                pulsar::Consumer c = *consumer;
+                self->startReceiveLoop(c, segmentId);
+                promise.setValue(c);
+            } else {

Review Comment:
   **[BUG] A transient segment subscribe failure can leave an active segment unconsumed indefinitely**
   
   On failure the segment is evicted from `segmentConsumers_` so a later reconcile can retry it — but off the first-layout path (line 138) nothing schedules that retry. `onLayoutChange` runs only when the broker pushes a DAG update.
   
   A reconnect does re-push the layout, so this self-heals in that case. What it does not survive is the common one: a healthy watch, a stable DAG, and a single segment whose subscribe failed on a transient lookup or connection error.
   
   Failure scenario: one of four active segments fails to subscribe during a layout change. The other three attach and the consumer looks healthy. The DAG does not change again for hours. Everything published to that fourth segment is never delivered, with nothing in the API surfacing the gap.
   
   A bounded retry with backoff for evicted segments would cover it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org