You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "JHSUYU (via GitHub)" <gi...@apache.org> on 2026/03/22 02:13:53 UTC
[PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
JHSUYU opened a new pull request, #8364:
URL: https://github.com/apache/hadoop/pull/8364
### Description of PR
Jira: [HDFS-17897](https://issues.apache.org/jira/browse/HDFS-17897)
HDFS-12931 added handling for `InvalidEncryptionKeyException` in `ReplicatedFileChecksumComputer.checksumBlock()` but missed the parallel striped file path `StripedFileNonStripedChecksumComputer.checksumBlockGroup()`.
Both paths call `DFSClient.connectToDN()`, which performs a SASL handshake using a cached `DataEncryptionKey` (DEK). When the DEK references a `BlockKey` that has been removed from the DataNode (there is
a time gap when the DataNode isn't updated with the new keys after key rotation, as described in [HDFS-12931](https://issues.apache.org/jira/browse/HDFS-12931)), the handshake fails with `InvalidEncryptionKeyException`.
In the replicated path, this exception is caught, `clearDataEncryptionKey()` is called to invalidate the cached DEK, and the block is retried. In the striped path, the exception falls through to the
generic `catch (IOException)` block, which only logs a warning. The stale DEK is never cleared, so every DataNode in the block group fails with the same error. The operation fails permanently — even
user-level retries will reuse the same stale cached DEK.
**Fix:** Add `catch (InvalidEncryptionKeyException)` in `checksumBlockGroup()`, mirroring the existing handling in `checksumBlock()`.
### How was this patch tested?
- Added `testStripedFileChecksumWithInvalidEncryptionKey` in `TestEncryptedTransfer`, which creates an EC file, invalidates the encryption key on all DataNodes, and verifies that `getFileChecksum()` succeeds by catching the exception and refreshing the key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8364:
URL: https://github.com/apache/hadoop/pull/8364#issuecomment-4130380772
How many are they? If it's less than, say, 5, let's group them into one PR.
If not, let's open multiple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "JHSUYU (via GitHub)" <gi...@apache.org>.
JHSUYU commented on PR #8364:
URL: https://github.com/apache/hadoop/pull/8364#issuecomment-4113540450
Hi @jojochuang, I noticed that you previously created an issue on JIRA (https://issues.apache.org/jira/browse/HDFS-16136), but nobody took it. I observed that this issue still occurs in other features, as shown in this PR. The HDFS fault tolerance mechanisms could not even mask the data encryption key error in these cases, causing applications to fail. I am very willing to fix all such issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "JHSUYU (via GitHub)" <gi...@apache.org>.
JHSUYU commented on PR #8364:
URL: https://github.com/apache/hadoop/pull/8364#issuecomment-4130505723
@jojochuang I found 7 places in total, including Balancer.Dispatcher, BlockDispatcher, StripedBlockWriter, DataNode, DataXceiver, StripedBlockReader, and NamenodeFsck. Do you think I should open 7 PRs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "JHSUYU (via GitHub)" <gi...@apache.org>.
JHSUYU commented on PR #8364:
URL: https://github.com/apache/hadoop/pull/8364#issuecomment-4115007270
> The fix makes sense. However, the test passes even if I comment out the fix.
>
> Also, we have this bad bad of always write system test instead of unit tests. That is a bigger problem though not expected to fix in this PR.
Thanks @jojochuang for the review! I'll update the test to make sure it actually fails without the fix. Later I will fix the remaining issues mentioned in HDFS-16136, it would be great if you could help review them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8364:
URL: https://github.com/apache/hadoop/pull/8364#issuecomment-4123992321
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 50m 34s | | trunk passed |
| +1 :green_heart: | compile | 6m 11s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 6m 28s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 2m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 32s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 2m 34s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 8m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 41s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 45s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 20s | | the patch passed |
| +1 :green_heart: | compile | 5m 27s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javac | 5m 27s | | the patch passed |
| +1 :green_heart: | compile | 6m 5s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | javac | 6m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 45s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 7m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 37s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 263m 54s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 454m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8364/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8364 |
| JIRA Issue | HDFS-17897 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9149184523f8 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d3e03c9430e527d8f20c2045d93193ecdf59ab3c |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8364/2/testReport/ |
| Max. process+thread count | 2236 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8364/2/console |
| versions | git=2.43.0 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8364:
URL: https://github.com/apache/hadoop/pull/8364#issuecomment-4124009532
Merged. Thanks @JHSUYU
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #8364:
URL: https://github.com/apache/hadoop/pull/8364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "JHSUYU (via GitHub)" <gi...@apache.org>.
JHSUYU commented on PR #8364:
URL: https://github.com/apache/hadoop/pull/8364#issuecomment-4106474918
@pan3793 Dear developer, could you take a look when you get a chance? Happy to update the patch based on any feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HDFS-17897] Handle InvalidEncryptionKeyException during striped file checksum [hadoop]
Posted by "JHSUYU (via GitHub)" <gi...@apache.org>.
JHSUYU commented on PR #8364:
URL: https://github.com/apache/hadoop/pull/8364#issuecomment-4130328904
Hi @jojochuang, I am working on fixing all the remaining places for handling InvalidEncryptionKeyException. May I ask, do you suggest I create a separate PR for each place and merge them under one JIRA issue, or just submit them in one PR? I just want to avoid a PR storm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org