You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/03/08 19:34:00 UTC
[jira] [Commented] (HDFS-17845) DataStreamer QuotaExceededException should be thrown to Client
[ https://issues.apache.org/jira/browse/HDFS-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063925#comment-18063925 ]
ASF GitHub Bot commented on HDFS-17845:
---------------------------------------
balodesecurity opened a new pull request, #8321:
URL: https://github.com/apache/hadoop/pull/8321
## Summary
Fixes two related problems when a `QuotaExceededException` (e.g. disk-space quota) is thrown during `DataStreamer.addBlock()`:
1. **Silent swallow / no WARN log** — the exception was caught and logged only at `DEBUG` level, making quota failures invisible in production logs.
2. **Exception never reaches the client** — because the namenode returns no datanode list when quota is exceeded, the output stream is never created and `lastException` is never surfaced, leaving the client blocked indefinitely.
### Changes
**`DataStreamer.java`**
- Remove the `QuotaExceededException`-specific `DEBUG` branch; all DataStreamer exceptions now emit a `WARN` log.
- After setting `lastException`, set `streamerClosed = true` and `notifyAll()` on `dataQueue` for `QuotaExceededException` so the client thread is unblocked immediately.
**`TestAbandonBlock.java`**
- Add `testQuotaExceptionPropagatedToClient`: sets a 1-byte space quota on a directory, appends enough bytes to trigger a second `addBlock`, and asserts that:
- `DSQuotaExceededException` is propagated to the caller.
- A `WARN`-level `"DataStreamer Exception"` log entry is emitted.
## Test plan
- [ ] `TestAbandonBlock#testQuotaExceptionPropagatedToClient` passes locally ✅
- [ ] Full module build passes (`mvn package ... -DskipTests`) ✅
> DataStreamer QuotaExceededException should be thrown to Client
> --------------------------------------------------------------
>
> Key: HDFS-17845
> URL: https://issues.apache.org/jira/browse/HDFS-17845
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Reporter: Karthik Palanisamy
> Priority: Major
>
> When a {{QuotaExceededException}} happens during {{{}addBlock{}}}, nameNode throws back to client but {{DataStreamer}} client logs this as {{DEBUG}} instead of {{WARN.}}
> {{[https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L857]}}
> Hard to troubleshoot them.
> Also, for any DataStreamer exception, we set it in lastException.set(e) and close the stream afterward. This lastException thrown in all the scenarios (when the stream is created with right datanode list). If any error during stream then it retries and send back to client. But in case of QuotaExceededException, the namenode didn’t return any datanode list, so the output stream was never created and also we missed throwing the exception to the client.
> [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L862]
> [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L876]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org