You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "GGraziadei (via GitHub)" <gi...@apache.org> on 2026/05/22 11:36:43 UTC
[PR] docker compose dev cluster (storm)
GGraziadei opened a new pull request, #8706:
URL: https://github.com/apache/storm/pull/8706
## What is the purpose of the change
This PR introduces a repeatable, Docker-based distributed Storm dev cluster designed for realistic benchmarking `storm-perf` on a local machine. It provisions a complete environment, including Nimbus, ZooKeeper, and two Supervisor, forcing inter-worker traffic across the network to trigger true serialization overhead.
Backed by a full observability stack (Prometheus and Grafana), the setup provides granular, per-task tracking via Storm Metrics v2. Additionally, it includes a `netsim.sh` utility to inject controlled network latency and jitter, allowing developers to easily stress-test topology resilience and analyze bottlenecks under degraded network conditions.
## How was the change tested
I verified the environment by executing the benchmark smoke test outlined in the README.md, running the `FileReadWordCountTopo` topology for 120 seconds across two workers on separate supervisors.
Smoke testing successfully validated the baseline performance and the replication of bottlenecks. Injecting typical datacenter network conditions (3 ms latency, 1 ms jitter) caused average complete latency to rise from 390 ms to 446 ms; this induced back-pressure safely reduced total tuple throughput from 40.93M to 36.25M without dropping packets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docker compose local (dev) distributed Storm cluster with full observability and network simulation (storm)
Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on code in PR #8706:
URL: https://github.com/apache/storm/pull/8706#discussion_r3311365913
##########
dev-tools/cluster/prepare-extlib.sh:
##########
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
Review Comment:
I totally agree. I'm moving this logic inside the `build-image.sh ` script and adding a flag to disable prepare-extlib. This will be useful for developers who don't need to touch the metrics logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docker compose local (dev) distributed Storm cluster with full observability and network simulation (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu commented on PR #8706:
URL: https://github.com/apache/storm/pull/8706#issuecomment-4583217071
Thanks for the contribution @GGraziadei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docker compose local (dev) distributed Storm cluster with full observability and network simulation (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu merged PR #8706:
URL: https://github.com/apache/storm/pull/8706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docker compose local (dev) distributed Storm cluster with full observability and network simulation (storm)
Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on PR #8706:
URL: https://github.com/apache/storm/pull/8706#issuecomment-4555349213
Thank you for the review. I addressed all the comments, and I have just pushed the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docker compose local (dev) distributed Storm cluster with full observability and network simulation (storm)
Posted by "GGraziadei (via GitHub)" <gi...@apache.org>.
GGraziadei commented on PR #8706:
URL: https://github.com/apache/storm/pull/8706#issuecomment-4532984646
Thanks for the detailed review and the helpful insights!
I have applied all the requested changes.
Added the exclusion for Grafana json dashboards in the root pom.xml and verified locally with mvn apache-rat:check -Prat.
Removed the references and configurations related to the unmerged tuple compression feature (apologies for the distraction and mixing the PR contents!).
Bound all published ports to 127.0.0.1 in docker-compose.yml to prevent LAN exposure.
Updated the README to state the Linux/macOS/WSL2 requirement explicitly, and resolved the hardcoded container names in netsim.sh, fixed the versioning fallback in prepare-extlib.sh, and added the missing trailing newline for a json file.
Everything is now pushed and ready for another look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docker compose local (dev) distributed Storm cluster with full observability and network simulation (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu commented on code in PR #8706:
URL: https://github.com/apache/storm/pull/8706#discussion_r3307416993
##########
dev-tools/cluster/prepare-extlib.sh:
##########
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
Review Comment:
can we add the execution of this script to build-image.sh ? It would be one less explicit step in setting up the environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docker compose local (dev) distributed Storm cluster with full observability and network simulation (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu commented on code in PR #8706:
URL: https://github.com/apache/storm/pull/8706#discussion_r3307378963
##########
dev-tools/cluster/netsim.sh:
##########
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Simulate network delay/jitter/loss on the worker hosts (supervisor containers)
+# using Linux tc/netem. The Storm image has no `tc`, so we inject the qdisc from
+# a throwaway helper container that shares each supervisor's network namespace
+# (--net container:...) -- no image rebuild, no compose change required.
+#
+# netem applies to *egress* on each supervisor's eth0, so it shapes ALL traffic
+# leaving that container (inter-worker tuples, but also heartbeats to Nimbus/ZK).
+# Keep the delay moderate (<= ~150ms) so heartbeats don't time out. With both
Review Comment:
can we echo a warning if supplied delay > 150?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org