You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "ErnestasVaiciukeviciusAdform (via GitHub)" <gi...@apache.org> on 2026/10/16 12:53:16 UTC

[I] `topology.upstream.feedback.enable` floods logs with "No remote destination available for task" and drops cross-worker feedback (storm)

ErnestasVaiciukeviciusAdform opened a new issue, #9174:
URL: https://github.com/apache/storm/issues/9174

   ## Summary
   When `topology.upstream.feedback.enable=true`, workers emit large volumes of
   
   ```
   WARN  o.a.s.utils.TransferDrainer - No remote destination available for task <id>
   ```
   
   at WARN level. The warning is not merely cosmetic: the upstream-feedback tuple that triggered it is **dropped**. As a result, upstream feedback is only ever delivered to tasks that live on the *same* worker; feedback destined for upstream tasks on *other* workers never arrives.
   
   Whether a given worker logs the warning (and drops feedback) depends entirely on **task placement**, which makes the symptom intermittent and misleading (see "Misleading symptom" below).
   
   ## Affected versions
   - Reproduced on `v3.0.0`.
   - Present on current `master` (the relevant code in `WorkerState.workerOutboundTasks()` is unchanged).
   
   ## Environment
   - Any topology running with `topology.upstream.feedback.enable=true` across **more than one worker** where at least one worker does **not** host a downstream consumer of a given upstream component.
   
   ## Steps to reproduce
   1. Submit any multi-component topology (e.g. `spout -> bolt1 -> bolt2`) with `topology.upstream.feedback.enable=true` (and `topology.stats.ewma.enable=true`, which feedback requires).
   2. Request enough workers that the components are spread out, so some worker hosts upstream tasks of a component but none of that component's downstream consumers.
   3. Observe the worker logs.
   
   ## Expected behaviour
   - Upstream feedback tuples are delivered to their target upstream tasks, including tasks on remote workers.
   - No WARN spam during normal operation.
   
   ## Actual behaviour
   - `TransferDrainer.send` cannot resolve the destination task and logs `No remote destination available for task {}` for every remote upstream-feedback recipient, on every feedback tick.
   - The corresponding feedback tuples are dropped, so remote upstream tasks never receive feedback.
   
   ## Root cause
   Upstream feedback is sent directly to **upstream (source) component task ids** via `Task.sendUnanchoredFeedback`, bypassing stream grouping (`Executor.sendUpstreamFeedback` → recipients from `Executor.computeUpstreamTaskIds`, `Executor.java:603,642`).
   
   A worker's routable remote destinations (`WorkerState.cachedTaskToNodePort`) and its remote connections are built **only** from `outboundTasks`, which `WorkerState.workerOutboundTasks()` computes as *the tasks of the downstream-target components of the locally-hosted tasks* (`WorkerState.java` `workerOutboundTasks`, consumed at `WorkerState.java:429`).
   
   `addUpstreamFeedback` (`StormCommon.java:363`) declares `FEEDBACK_STREAM_ID` only as an **output** stream on each component — it creates **no input/grouping edge**. Consequently, upstream feedback recipients are never part of `outboundTasks`. When such a recipient is remote, `taskToNode.get(taskId)` is `null` in `TransferDrainer.send` (`TransferDrainer.java:72`), the tuple is dropped, and the warning is logged.
   
   ## Misleading symptom (important)
   `outboundTasks` membership is decided **per component**: if a component is a downstream target of *any* local task, **all** of that component's tasks (topology-wide) become routable.
   
   Because of this:
   - A worker produces **no warning** for feedback to an upstream component `X` whenever `X` also happens to be a **downstream target** of some task co-located on that same worker — the feedback rides the already-established connection and looks healthy.
   - The *same* feedback edge on a *different* worker (one hosting upstream tasks of `X` but none of `X`'s downstream consumers) logs the warning and drops the tuple.
   
   Therefore **"no warnings" does not mean feedback is being delivered** — it only means the scheduler happened to co-locate a downstream consumer of `X`. In particular, if every worker hosts at least one task of every component, the warnings disappear entirely even though the underlying routing gap still exists. This placement dependence can easily mislead operators into believing the feature works.
   
   ## Proposed fix
   When upstream feedback is enabled, also include the non-system **source (upstream) components** of the locally-hosted components when building `outboundTasks`, mirroring `Executor.computeUpstreamTaskIds()` at worker granularity. This makes every feedback recipient (bolt or spout) a routable remote destination with a connection, delivering feedback correctly and eliminating the spurious warning — independent of task placement.
   
   The fix is limited to `WorkerState.workerOutboundTasks()`; behaviour only changes when `topology.upstream.feedback.enable=true` (off by default).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] `topology.upstream.feedback.enable` floods logs with "No remote destination available for task" and drops cross-worker feedback (storm)

Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei closed issue #9174: `topology.upstream.feedback.enable` floods logs with "No remote destination available for task" and drops cross-worker feedback
URL: https://github.com/apache/storm/issues/9174


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org