You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "zouhuajian (via GitHub)" <gi...@apache.org> on 2026/08/30 08:56:27 UTC
[PR] HDFS-17970. Exclude failed EC checksum target from reconstruction sources [hadoop]
zouhuajian opened a new pull request, #8709:
URL: https://github.com/apache/hadoop/pull/8709
### Description of PR
This PR fixes [HDFS-17970](https://issues.apache.org/jira/browse/HDFS-17970).
During striped block checksum reconstruction, the DataNode that failed the original checksum request could still be selected as a reconstruction source. This may cause reconstruction to read from the same failed target and prevent the file checksum from being calculated.
This change:
- excludes every location belonging to the failed checksum target from the reconstruction source candidates;
- keeps source block indices aligned with their corresponding DataNode locations during source selection;
- preserves valid non-target replicas when selecting the required sources;
- adds a deterministic regression test that triggers the checksum reconstruction fallback with an invalid child block token.
Contains content generated by OpenAI Codex.
### How was this patch tested?
- Ran the focused regression test: `TestFileChecksum#testStripedFileChecksumReconstructionExcludesFailedTarget` (`2` tests passed).
- Ran the complete `TestFileChecksum` test class (`72` tests passed).
- Ran `git diff --check`.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? *Note: Automated CI testing doesn't cover all cases so manual testing with cloud storage is still required.*
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
The unchecked object storage, dependency, and license-file items are not applicable to this change.
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by OpenAI Codex."
- [ ] My use of AI contributions follows the ASF legal policy [https://www.apache.org/legal/generative-tooling.html](https://www.apache.org/legal/generative-tooling.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17970. Exclude failed EC checksum target from reconstruction sources [hadoop]
Posted by "qijiale76 (via GitHub)" <gi...@apache.org>.
qijiale76 commented on PR #8709:
URL: https://github.com/apache/hadoop/pull/8709#issuecomment-5748647870
Hi @zouhuajian , thanks for this fix. We are evaluating HDFS-17970 for production and have two questions:
1. Have you deployed this patch in production? If so, have you observed any issues with EC checksum reconstruction, particularly during DataNode decommissioning or when duplicate internal-block replicas are present?
2. Have you considered duplicate replicas being counted as independent sources during reconstruction fallback?
We reproduced the following case on trunk `3f440d242312` plus this PR’s commit `ac4a2ce42984`, using JDK 17.
For RS-6-3, reconstructing index `0`, the source indices are ordered as the patch allows:
```text
[1, 2, 3, 4, 5, 6, 1, 6]
```
If opening the first index-6 reader fails, `StripedReader.initReaders()` selects `[1, 2, 3, 4, 5, 1]` and stops after six successful readers. The healthy index-6 replica at the end is never tried.
`getInputBuffers()` then places both index-1 buffers into the same slot, leaving only five distinct inputs. The Java RS decoder throws:
```text
No enough valid inputs are provided, not recoverable
```
Our targeted test uses the actual trunk `StripedReader` selection logic and Java RS decoder, with mocked readers to inject the connection failure. The control case with all primary sources available decodes successfully. The PR’s existing MD5 and Composite CRC regression tests also pass. We have not reproduced this with real DataNode failures in a cluster.
The underlying reader behavior predates this PR. However, retaining duplicate replicas as fallbacks appears to require counting distinct internal-block indices, both during initialization and when replacing failed or slow reads.
Does this match your understanding? Have you encountered this case, or is there an existing issue or follow-up patch? Please let us know if there is an invariant we have missed that would prevent it in practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17970. Exclude failed EC checksum target from reconstruction sources [hadoop]
Posted by "qijiale76 (via GitHub)" <gi...@apache.org>.
qijiale76 commented on PR #8709:
URL: https://github.com/apache/hadoop/pull/8709#issuecomment-5755278036
@zouhuajian Thanks for the clarification and for updating the patch.
We have now reproduced the HDFS-17970 failure mode in production with a diagnostics-only build. One representative case was:
```text
target=0
trigger=SocketTimeoutException
triggerMs=10021
triggerDn=DN-A
requiredSources=2
candidates=[0:0@DN-A, 1:6@DN-B, 2:7@DN-C, 3:8@DN-D, 4:1@DN-E]
selectedPositions=[1,0]
selectedIndices=[6,0]
selectedUnique=2
selectedTarget=true
inputIndices=[0,2,3,4,5,6]
targetInInputs=true
validator=erased=0,valid=[2,3,4,5,6]
error=HadoopIllegalArgumentException
```
This confirms the target/source overlap in production: after the child checksum request for index 0 times out, index 0 remains in the reconstruction source candidates and is selected again as an input, causing validation to fail with insufficient valid inputs for RS-6-3.
For our case, both versions of the patch would fix the problem, since both prevent the reconstruction target from being used as a source.
That said, I prefer the latest revision. Preserving the original source order and only excluding the target is a more focused fix for HDFS-17970 and avoids changing the existing fallback behavior. The duplicate-source counting issue is better handled separately in `StripedReader`.
The current patch looks correct to me and I think it is ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17970. Exclude failed EC checksum target from reconstruction sources [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8709:
URL: https://github.com/apache/hadoop/pull/8709#issuecomment-5758351343
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 4s | | trunk passed |
| +1 :green_heart: | compile | 1m 51s | | trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | compile | 1m 49s | | trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 4m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 4m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 261m 50s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not generate ASF License warnings. |
| | | 412m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8709 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5f6f789c479e 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 09e04a098cbf256cff562bcb3e35fd15954b9a4e |
| Default Java | Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12+8-1-24.04-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/3/testReport/ |
| Max. process+thread count | 2306 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/3/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17970. Exclude failed EC checksum target from reconstruction sources [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8709:
URL: https://github.com/apache/hadoop/pull/8709#issuecomment-5469819233
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 19m 46s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 50s | | trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | compile | 1m 49s | | trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 4m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 4m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 264m 21s | | hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 0m 55s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 438m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8709 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ae8fd63dc7b3 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2339461c0bdc3d30e3f3eb329c540eb3d7d414ff |
| Default Java | Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12+8-1-24.04-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/1/testReport/ |
| Max. process+thread count | 2468 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8709/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17970. Exclude failed EC checksum target from reconstruction sources [hadoop]
Posted by "zouhuajian (via GitHub)" <gi...@apache.org>.
zouhuajian commented on PR #8709:
URL: https://github.com/apache/hadoop/pull/8709#issuecomment-5569917354
Hi @Hexiaoqiao , could you please take a look when you have time? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17970. Exclude failed EC checksum target from reconstruction sources [hadoop]
Posted by "zouhuajian (via GitHub)" <gi...@apache.org>.
zouhuajian commented on PR #8709:
URL: https://github.com/apache/hadoop/pull/8709#issuecomment-5754545252
> Hi @zouhuajian , thanks for this fix. We are evaluating [HDFS-17970](https://issues.apache.org/jira/browse/HDFS-17970) for production and have two questions:
>
> 1. Have you deployed this patch in production? If so, have you observed any issues with EC checksum reconstruction, particularly during DataNode decommissioning or when duplicate internal-block replicas are present?
> 2. Have you considered duplicate replicas being counted as independent sources during reconstruction fallback?
>
> We reproduced the following case on trunk `3f440d242312` plus this PR’s commit `ac4a2ce42984`, using JDK 17.
>
> For RS-6-3, reconstructing index `0`, the source indices are ordered as the patch allows:
>
> ```
> [1, 2, 3, 4, 5, 6, 1, 6]
> ```
>
> If opening the first index-6 reader fails, `StripedReader.initReaders()` selects `[1, 2, 3, 4, 5, 1]` and stops after six successful readers. The healthy index-6 replica at the end is never tried.
>
> `getInputBuffers()` then places both index-1 buffers into the same slot, leaving only five distinct inputs. The Java RS decoder throws:
>
> ```
> No enough valid inputs are provided, not recoverable
> ```
>
> Our targeted test uses the actual trunk `StripedReader` selection logic and Java RS decoder, with mocked readers to inject the connection failure. The control case with all primary sources available decodes successfully. The PR’s existing MD5 and Composite CRC regression tests also pass. We have not reproduced this with real DataNode failures in a cluster.
>
> The underlying reader behavior predates this PR. However, retaining duplicate replicas as fallbacks appears to require counting distinct internal-block indices, both during initialization and when replacing failed or slow reads.
>
> Does this match your understanding? Have you encountered this case, or is there an existing issue or follow-up patch? Please let us know if there is an invariant we have missed that would prevent it in practice.
Thanks for the detailed reproduction. I agree with your analysis, `StripedReader` counts successful readers rather than distinct internal block indices, so it can stop before collecting enough independent inputs, both during initialization and when replacing failed or slow reads.
The original goal of this PR is to prevent the checksum reconstruction target from also being selected as a source. The source reordering in the current patch only ensures that the initial candidates have distinct indices; it does not guarantee this after failures or timeouts. It can also change which fallback reads are attempted, so I think it goes beyond the scope of this fix.
I have prepared a local revision that removes the reordering and the additional source count check. It only excludes entries matching the target index, preserving the order of all remaining candidates, including duplicate replicas.
Your reproduction still applies after this change. I think the duplicate-source counting issue would be best addressed in a separate reader fix, with regression tests for both initialization and replacement reads. HDFS-14946 addressed the initial source ordering for NameNode scheduled reconstruction, but it does not ensure that the readers selected after connection failures or read timeouts correspond to distinct internal block indices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org