You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "joseluisll (via GitHub)" <gi...@apache.org> on 2026/09/07 20:14:15 UTC
[PR] HADOOP-19986. MiniDFSCluster.shutdownDataNodes() should signal all DataNodes before joining any [hadoop]
joseluisll opened a new pull request, #8723:
URL: https://github.com/apache/hadoop/pull/8723
### Description of PR
https://issues.apache.org/jira/browse/HADOOP-19986
`MiniDFSCluster.shutdownDataNodes()` tears DataNodes down strictly serially - stop one, join it, move to the next. The DataNodes not yet reached keep retrying a NameNode the test has already killed.
DataNodes in a single JVM share an `ipc.Client` through `ClientCache`, and therefore share its per-address `Connection` objects. A surviving DataNode's `BPServiceActor` holds a `Connection` monitor across its connect-retry sleeps in `handleConnectionFailure`, while an actor of the DataNode being joined sits BLOCKED on that same monitor in `Client.addCall`. `Thread.interrupt()` cannot dislodge a BLOCKED thread, so the `stop()` issued by `BlockPoolManager.shutDownAll` is ineffective and the join waits on scheduling luck: the holder releases and re-acquires roughly every 2s, and unfair monitors starve the blocked thread.
Measured on a 2-core runner: DataNode 2 took 165s to shut down, after which DN1 and DN0 finished in ~10ms. That overran the 180s timeout of `TestBalancerWithHANameNodes#testBalancerWithObserverWithFailedNode`, and runs that stayed under the deadline still burned ~115s in teardown.
The fix signals every DataNode before joining any, so the monitor holder aborts its sleep and releases it:
1. `BlockPoolManager#signalShutDownAll` - the stop-without-join half of the existing `shutDownAll`, which now delegates to it.
2. `DataNode#signalBlockPoolShutdown` - `@VisibleForTesting`, null-safe, signals every block pool service without waiting.
3. `MiniDFSCluster#shutdownDataNodes` - signals all DataNodes up front, then runs the existing per-DataNode shutdown loop unchanged.
`stop()` is idempotent, so the per-DataNode shutdown path behaves exactly as before; the only change is that the interrupts now all land before the first join.
### How was this patch tested?
20 runs of `TestBalancerWithHANameNodes#testBalancerWithObserverWithFailedNode` on a 2-core runner. Before: 4 of 20 anomalous (182.3s, 181.3s, 116.7s, 113.9s). After: 20 of 20 passed within 44-48s.
Full CI on the fork, all jobs green - `common`, `hdfs - other`, `hdfs - slow`, `hdfs-rbf`, `mr`, `other`, `yarn-server-rm` on Java 17, plus build-only on Java 21 and Java 25:
https://github.com/joseluisll/hadoop/actions/runs/34147776139
That run was on commit `1700e04b`; this branch has since been rebased onto current trunk with no change to the patch itself.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
Contains content generated by Claude Code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19986. MiniDFSCluster.shutdownDataNodes() should signal all DataNodes before joining any [hadoop]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao merged PR #8723:
URL: https://github.com/apache/hadoop/pull/8723
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19986. MiniDFSCluster.shutdownDataNodes() should signal all DataNodes before joining any [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8723:
URL: https://github.com/apache/hadoop/pull/8723#issuecomment-5578640018
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 20m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 9s | | trunk passed |
| +1 :green_heart: | compile | 1m 50s | | trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | compile | 1m 46s | | trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 4m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 4m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 258m 14s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 429m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8723/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8723 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 86cf0713d074 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b97f5f8aca56bca5e730754fb4f0affa61ff9006 |
| Default Java | Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12+8-1-24.04-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8723/1/testReport/ |
| Max. process+thread count | 2320 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8723/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19986. MiniDFSCluster.shutdownDataNodes() should signal all DataNodes before joining any [hadoop]
Posted by "joseluisll (via GitHub)" <gi...@apache.org>.
joseluisll commented on PR #8723:
URL: https://github.com/apache/hadoop/pull/8723#issuecomment-5579024588
@pan3793 @slfan1989 This PR is Green and ready to be reviewed. This one solves race conditions on minicluster when asked to be shutdown, it hanged on the closing datanodes. This surfaced because up to a month ago minicluster was not shutdown in the tests, leaked. I worked in the jiras to fix that. Properly closing the miniclusters and eliminating this race condition improves CI testing.
It is a prerrequisite for HADOOP-19979, that will fix 4 flaky conditions so that we reduce later the GHA exclude list, as explained on HADOOP-19981 [umbrella for all GHA reactivation candidates from excluded-tests.txt] and its subtasks [specific identified candidates for reactivation].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19986. MiniDFSCluster.shutdownDataNodes() should signal all DataNodes before joining any [hadoop]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao commented on PR #8723:
URL: https://github.com/apache/hadoop/pull/8723#issuecomment-5724998666
Committed to trunk. Thanks @joseluisll
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org