Re: [I] Decouple the control plane (system streams) from the data plane on executor receive queues (storm)

["GGraziadei (via GitHub)" <gi...@apache.org>]

GGraziadei commented on issue #8816:
URL: https://github.com/apache/storm/issues/8816#issuecomment-4879907659

   Hi @rzo1, thanks for the deep dive and the proposal.
   Your design is a major improvement over the initial approach. 
   Embedding the control lane directly within `JCQueue` is definitely a winning move: it drastically reduces the refactoring blast radius (avoiding changes to queue mapping, backpressure tracking, and metrics wiring) while elegantly solving the three failure modes you highlighted, especially the blocking `userTimer` thread.
   You are also absolutely right about the ackers. Ack traffic is strictly tied to data plane volume (being a function of the graph path size multiplied by the number of tuples). Routing them into the control lane would have introduced a concrete risk of unbounded growth or breaking at-least-once semantics. My initial classification suffered from this bias, and I fully agree with adopting your explicit, restricted whitelist (`__flush`, `__tick`, etc.) while keeping everything else on the data path.
   
   I have just opened a draft PR #8892  with the core structure of this solution. I will soon add the benchmarck report.
   Cheers!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org