You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "boroknagyz (via GitHub)" <gi...@apache.org> on 2026/06/22 15:41:40 UTC
[PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
boroknagyz opened a new pull request, #8561:
URL: https://github.com/apache/hadoop/pull/8561
Adds support for opening HDFS files with pre-fetched block locations, eliminating redundant NameNode RPCs when callers already have cached block metadata.
Changes:
- DFSClient.open() overload accepting LocatedBlocks
- DistributedFileSystem.openFileWithOptions() override that extracts LocatedBlocks from HdfsLocatedFileStatus passed via withFileStatus()
- DistributedFileSystem.cloneDataInputStream() to create independent file handles reusing block locations from an existing stream
- libhdfs hdfsCloneFile() C API for cloning file handles via JNI
- Integration tests verifying no NameNode RPC on clone/open with status
Contains content generated by Claude Opus 4.6
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Adds support for opening HDFS files with pre-fetched block locations, eliminating redundant NameNode RPCs when callers already have cached block metadata.
### How was this patch tested?
Unit tests.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4825087504
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 21s | | https://github.com/apache/hadoop/pull/8561 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/6/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4806674576
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 55s | | trunk passed |
| +1 :green_heart: | compile | 5m 28s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 5m 54s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 2m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 31s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 48s | | branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 28m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 42s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 27s | | the patch passed |
| +1 :green_heart: | compile | 5m 5s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | cc | 5m 5s | | the patch passed |
| +1 :green_heart: | golang | 5m 5s | | the patch passed |
| +1 :green_heart: | javac | 5m 5s | | the patch passed |
| +1 :green_heart: | compile | 5m 21s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | cc | 5m 21s | | the patch passed |
| +1 :green_heart: | golang | 5m 21s | | the patch passed |
| +1 :green_heart: | javac | 5m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 44s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 20s | | hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 27m 18s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 40s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 217m 41s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 9m 32s | | hadoop-hdfs-native-client in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 395m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux b1d7ddb960f7 5.15.0-177-generic #187-Ubuntu SMP Sat Apr 11 22:54:33 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 97f5a2df29c91cde9af42bbe4b654b274232bc05 |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/testReport/ |
| Max. process+thread count | 4091 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4807609539
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 23s | | https://github.com/apache/hadoop/pull/8561 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/5/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4807591473
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 24s | | https://github.com/apache/hadoop/pull/8561 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/5/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4773965649
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 14m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 46m 44s | | trunk passed |
| +1 :green_heart: | compile | 6m 57s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | compile | 7m 28s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | checkstyle | 3m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 31s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 6m 32s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 1m 51s | | branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 32m 1s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 2m 38s | [/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-native-client.txt) | hadoop-hdfs-native-client in the patch failed. |
| -1 :x: | compile | 4m 8s | [/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-21.0.11+10-LTS.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-21.0.11+10-LTS.txt) | hadoop-hdfs-project in the patch failed with JDK Red Hat, Inc.-21.0.11+10-LTS. |
| -1 :x: | cc | 4m 8s | [/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-21.0.11+10-LTS.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-21.0.11+10-LTS.txt) | hadoop-hdfs-project in the patch failed with JDK Red Hat, Inc.-21.0.11+10-LTS. |
| -1 :x: | golang | 4m 8s | [/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-21.0.11+10-LTS.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-21.0.11+10-LTS.txt) | hadoop-hdfs-project in the patch failed with JDK Red Hat, Inc.-21.0.11+10-LTS. |
| -1 :x: | javac | 4m 8s | [/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-21.0.11+10-LTS.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-21.0.11+10-LTS.txt) | hadoop-hdfs-project in the patch failed with JDK Red Hat, Inc.-21.0.11+10-LTS. |
| -1 :x: | compile | 3m 42s | [/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-17.0.19+10-LTS.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-17.0.19+10-LTS.txt) | hadoop-hdfs-project in the patch failed with JDK Red Hat, Inc.-17.0.19+10-LTS. |
| -1 :x: | cc | 3m 42s | [/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-17.0.19+10-LTS.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-17.0.19+10-LTS.txt) | hadoop-hdfs-project in the patch failed with JDK Red Hat, Inc.-17.0.19+10-LTS. |
| -1 :x: | golang | 3m 42s | [/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-17.0.19+10-LTS.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-17.0.19+10-LTS.txt) | hadoop-hdfs-project in the patch failed with JDK Red Hat, Inc.-17.0.19+10-LTS. |
| -1 :x: | javac | 3m 42s | [/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-17.0.19+10-LTS.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project-jdkRedHat,Inc.-17.0.19+10-LTS.txt) | hadoop-hdfs-project in the patch failed with JDK Red Hat, Inc.-17.0.19+10-LTS. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 58s | [/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/results-checkstyle-hadoop-hdfs-project.txt) | hadoop-hdfs-project: The patch generated 2 new + 55 unchanged - 0 fixed = 57 total (was 55) |
| +1 :green_heart: | mvnsite | 3m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 38s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 2m 38s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 0m 34s | | hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 28m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 4s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 222m 33s | | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 3m 13s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt) | hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not generate ASF License warnings. |
| | | 444m 45s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux e35c49f37d45 5.15.0-177-generic #187-Ubuntu SMP Sat Apr 11 22:54:33 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c72419ef6f4dd356540e8c2980884c68e4762fef |
| Default Java | Red Hat, Inc.-17.0.19+10-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.11.0.10-1.el8_10.x86_64:Red Hat, Inc.-21.0.11+10-LTS /usr/lib/jvm/java-17-openjdk-17.0.19.0.10-1.el8_10.x86_64:Red Hat, Inc.-17.0.19+10-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/testReport/ |
| Max. process+thread count | 3490 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/console |
| versions | git=2.43.7 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran merged PR #8561:
URL: https://github.com/apache/hadoop/pull/8561
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4775926449
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 12m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 31s | | trunk passed |
| +1 :green_heart: | compile | 5m 39s | | trunk passed |
| +1 :green_heart: | checkstyle | 2m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 28s | | trunk passed |
| +0 :ok: | spotbugs | 0m 49s | | branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 28m 44s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 42s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 46s | [/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-native-client.txt) | hadoop-hdfs-native-client in the patch failed. |
| -1 :x: | compile | 3m 10s | [/patch-compile-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project.txt) | hadoop-hdfs-project in the patch failed. |
| -1 :x: | cc | 3m 10s | [/patch-compile-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project.txt) | hadoop-hdfs-project in the patch failed. |
| -1 :x: | golang | 3m 10s | [/patch-compile-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project.txt) | hadoop-hdfs-project in the patch failed. |
| -1 :x: | javac | 3m 10s | [/patch-compile-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-compile-hadoop-hdfs-project.txt) | hadoop-hdfs-project in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 39s | [/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/results-checkstyle-hadoop-hdfs-project.txt) | hadoop-hdfs-project: The patch generated 2 new + 55 unchanged - 0 fixed = 57 total (was 55) |
| +1 :green_heart: | mvnsite | 2m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 55s | | the patch passed |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 27m 20s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 38s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 218m 53s | | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 1m 59s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt) | hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 379m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 7e0243579d96 5.15.0-177-generic #187-Ubuntu SMP Sat Apr 11 22:54:33 UTC 2026 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c72419ef6f4dd356540e8c2980884c68e4762fef |
| Default Java | Debian-25.0.3+9-2-deb13u1-Debian |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/testReport/ |
| Max. process+thread count | 3317 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/1/console |
| versions | git=2.47.3 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4804911185
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 12m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 21s | | trunk passed |
| +1 :green_heart: | compile | 5m 31s | | trunk passed |
| +1 :green_heart: | checkstyle | 2m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed |
| +0 :ok: | spotbugs | 0m 49s | | branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 28m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 24s | | the patch passed |
| +1 :green_heart: | compile | 5m 3s | | the patch passed |
| +1 :green_heart: | cc | 5m 3s | | the patch passed |
| +1 :green_heart: | golang | 5m 3s | | the patch passed |
| +1 :green_heart: | javac | 5m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 27m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 41s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 221m 5s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 9m 24s | | hadoop-hdfs-native-client in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 392m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux cf8c8e4d2fde 5.15.0-177-generic #187-Ubuntu SMP Sat Apr 11 22:54:33 UTC 2026 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 97f5a2df29c91cde9af42bbe4b654b274232bc05 |
| Default Java | Debian-25.0.3+9-2-deb13u1-Debian |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/testReport/ |
| Max. process+thread count | 3249 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/console |
| versions | git=2.47.3 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "boroknagyz (via GitHub)" <gi...@apache.org>.
boroknagyz commented on code in PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#discussion_r3458825183
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##########
@@ -378,6 +383,72 @@ public FSDataInputStream open(PathHandle fd, int bufferSize)
return dfs.createWrappedInputStream(dfsis);
}
+ @Override
+ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
+ final Path path,
+ final OpenFileParameters parameters) throws IOException {
+ AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
+ parameters.getMandatoryKeys(),
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS,
+ "for " + path);
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ final Path absF = fixRelativePart(path);
+ return LambdaUtils.eval(new CompletableFuture<>(), () -> {
+ LocatedBlocks locatedBlocks =
+ getLocatedBlocksFromStatus(parameters.getStatus());
+ final DFSInputStream dfsis;
+ if (locatedBlocks != null) {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum, locatedBlocks);
+ } else {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum);
+ }
+ return dfs.createWrappedInputStream(dfsis);
+ });
+ }
+
+ private static LocatedBlocks getLocatedBlocksFromStatus(FileStatus status) {
+ if (status instanceof HdfsLocatedFileStatus) {
+ return ((HdfsLocatedFileStatus) status).getLocatedBlocks();
+ }
+ return null;
+ }
+
+ /**
+ * Create a new input stream for the same file as an existing stream,
+ * reusing its cached block locations to avoid a NameNode RPC.
+ * The returned stream is independent (its own position, buffers, etc.)
+ * but shares the same block location metadata.
+ *
+ * @param existing an open input stream obtained from this filesystem
+ * @return a new independent input stream for the same file
+ * @throws IOException if the stream cannot be cloned
+ */
+ public FSDataInputStream cloneDataInputStream(FSDataInputStream existing)
+ throws IOException {
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ InputStream wrapped = existing.getWrappedStream();
+ DFSInputStream dfsis;
+ if (wrapped instanceof DFSInputStream) {
+ dfsis = (DFSInputStream) wrapped;
+ } else if (wrapped instanceof org.apache.hadoop.crypto.CryptoInputStream) {
Review Comment:
Done.
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestOpenFileWithLocatedBlocks.java:
##########
@@ -0,0 +1,475 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import static org.apache.hadoop.test.MetricsAsserts.assertCounter;
+import static org.apache.hadoop.test.MetricsAsserts.getMetrics;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.crypto.CryptoInputStream;
+import org.apache.hadoop.crypto.key.JavaKeyStoreProvider;
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hdfs.client.CreateEncryptionZoneFlag;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests that opening a file with pre-fetched block locations via the
+ * {@code openFile().withFileStatus()} builder API skips the NameNode RPC
+ * for block locations.
+ */
+public class TestOpenFileWithLocatedBlocks {
+ private static final String NN_METRICS = "NameNodeActivity";
+ private static final int BLOCK_SIZE = 1024;
+ private static final short REPLICATION = 1;
+ private static final String TEST_KEY = "test_key";
+ private static final EnumSet<CreateEncryptionZoneFlag> NO_TRASH =
+ EnumSet.of(CreateEncryptionZoneFlag.NO_TRASH);
+
+ private MiniDFSCluster cluster;
+ private DistributedFileSystem fs;
+ private Configuration conf;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ conf = new HdfsConfiguration();
+ conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCK_SIZE);
+ FileSystemTestHelper fsHelper = new FileSystemTestHelper();
+ File testRootDir = new File(fsHelper.getTestRootDir()).getAbsoluteFile();
+ conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH,
+ JavaKeyStoreProvider.SCHEME_NAME + "://file" +
+ new Path(testRootDir.toString(), "test.jks").toUri());
+ cluster = new MiniDFSCluster.Builder(conf)
+ .numDataNodes(1)
+ .build();
+ cluster.waitActive();
+ fs = cluster.getFileSystem();
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (cluster != null) {
+ cluster.shutdown();
+ cluster = null;
+ }
+ }
+
+ private Path createFile(String name, byte[] data) throws IOException {
Review Comment:
Thanks, switched to ContractTestUtils.
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestOpenFileWithLocatedBlocks.java:
##########
@@ -0,0 +1,475 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import static org.apache.hadoop.test.MetricsAsserts.assertCounter;
+import static org.apache.hadoop.test.MetricsAsserts.getMetrics;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.crypto.CryptoInputStream;
+import org.apache.hadoop.crypto.key.JavaKeyStoreProvider;
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hdfs.client.CreateEncryptionZoneFlag;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests that opening a file with pre-fetched block locations via the
+ * {@code openFile().withFileStatus()} builder API skips the NameNode RPC
+ * for block locations.
+ */
+public class TestOpenFileWithLocatedBlocks {
+ private static final String NN_METRICS = "NameNodeActivity";
+ private static final int BLOCK_SIZE = 1024;
+ private static final short REPLICATION = 1;
+ private static final String TEST_KEY = "test_key";
+ private static final EnumSet<CreateEncryptionZoneFlag> NO_TRASH =
+ EnumSet.of(CreateEncryptionZoneFlag.NO_TRASH);
+
+ private MiniDFSCluster cluster;
+ private DistributedFileSystem fs;
+ private Configuration conf;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ conf = new HdfsConfiguration();
+ conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCK_SIZE);
+ FileSystemTestHelper fsHelper = new FileSystemTestHelper();
+ File testRootDir = new File(fsHelper.getTestRootDir()).getAbsoluteFile();
+ conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH,
+ JavaKeyStoreProvider.SCHEME_NAME + "://file" +
+ new Path(testRootDir.toString(), "test.jks").toUri());
+ cluster = new MiniDFSCluster.Builder(conf)
Review Comment:
Done.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##########
@@ -378,6 +383,72 @@ public FSDataInputStream open(PathHandle fd, int bufferSize)
return dfs.createWrappedInputStream(dfsis);
}
+ @Override
+ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
+ final Path path,
+ final OpenFileParameters parameters) throws IOException {
+ AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
+ parameters.getMandatoryKeys(),
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS,
+ "for " + path);
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ final Path absF = fixRelativePart(path);
+ return LambdaUtils.eval(new CompletableFuture<>(), () -> {
+ LocatedBlocks locatedBlocks =
+ getLocatedBlocksFromStatus(parameters.getStatus());
+ final DFSInputStream dfsis;
+ if (locatedBlocks != null) {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum, locatedBlocks);
+ } else {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum);
+ }
+ return dfs.createWrappedInputStream(dfsis);
+ });
+ }
+
+ private static LocatedBlocks getLocatedBlocksFromStatus(FileStatus status) {
+ if (status instanceof HdfsLocatedFileStatus) {
+ return ((HdfsLocatedFileStatus) status).getLocatedBlocks();
+ }
+ return null;
+ }
+
+ /**
+ * Create a new input stream for the same file as an existing stream,
+ * reusing its cached block locations to avoid a NameNode RPC.
+ * The returned stream is independent (its own position, buffers, etc.)
+ * but shares the same block location metadata.
+ *
+ * @param existing an open input stream obtained from this filesystem
+ * @return a new independent input stream for the same file
+ * @throws IOException if the stream cannot be cloned
+ */
+ public FSDataInputStream cloneDataInputStream(FSDataInputStream existing)
+ throws IOException {
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ InputStream wrapped = existing.getWrappedStream();
+ DFSInputStream dfsis;
+ if (wrapped instanceof DFSInputStream) {
+ dfsis = (DFSInputStream) wrapped;
+ } else if (wrapped instanceof org.apache.hadoop.crypto.CryptoInputStream) {
+ dfsis = (DFSInputStream)
+ ((org.apache.hadoop.crypto.CryptoInputStream) wrapped)
+ .getWrappedStream();
+ } else {
+ throw new IOException("Cannot clone: underlying stream is "
+ + wrapped.getClass().getName() + ", not a DFSInputStream");
+ }
+ LocatedBlocks locatedBlocks = dfsis.getLocatedBlocks();
+ String src = dfsis.getSrc();
+ DFSInputStream clone = dfs.open(src,
+ dfs.getConf().getIoBufferSize(), verifyChecksum, locatedBlocks);
+ return dfs.createWrappedInputStream(clone);
Review Comment:
Done.
##########
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include/hdfs/hdfs.h:
##########
@@ -685,11 +685,27 @@ extern "C" {
int hdfsCloseFile(hdfsFS fs, hdfsFile file);
- /**
- * hdfsExists - Checks if a given path exsits on the filesystem
+ /**
+ * hdfsCloneFile - Create a new independent file handle for the same file
+ * as an existing handle, reusing cached block locations to avoid a
+ * NameNode RPC. The cloned handle has its own read position and buffers.
+ * Only works for input (read) streams on a DistributedFileSystem.
+ *
+ * @param fs The configured filesystem handle.
+ * @param file An open input file handle to clone.
+ * @return Returns a new file handle on success, NULL on error.
+ * On error, errno will be set appropriately.
+ * The returned handle must be closed with hdfsCloseFile.
+ */
+ LIBHDFS_EXTERNAL
+ hdfsFile hdfsCloneFile(hdfsFS fs, hdfsFile file);
+
+
+ /**
+ * hdfsExists - Checks if a given path exsits on the filesystem
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#discussion_r3454963767
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##########
@@ -378,6 +383,72 @@ public FSDataInputStream open(PathHandle fd, int bufferSize)
return dfs.createWrappedInputStream(dfsis);
}
+ @Override
+ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
+ final Path path,
+ final OpenFileParameters parameters) throws IOException {
+ AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
+ parameters.getMandatoryKeys(),
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS,
+ "for " + path);
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ final Path absF = fixRelativePart(path);
+ return LambdaUtils.eval(new CompletableFuture<>(), () -> {
+ LocatedBlocks locatedBlocks =
+ getLocatedBlocksFromStatus(parameters.getStatus());
+ final DFSInputStream dfsis;
+ if (locatedBlocks != null) {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum, locatedBlocks);
+ } else {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum);
+ }
+ return dfs.createWrappedInputStream(dfsis);
+ });
+ }
+
+ private static LocatedBlocks getLocatedBlocksFromStatus(FileStatus status) {
+ if (status instanceof HdfsLocatedFileStatus) {
+ return ((HdfsLocatedFileStatus) status).getLocatedBlocks();
+ }
+ return null;
+ }
+
+ /**
+ * Create a new input stream for the same file as an existing stream,
+ * reusing its cached block locations to avoid a NameNode RPC.
+ * The returned stream is independent (its own position, buffers, etc.)
+ * but shares the same block location metadata.
+ *
+ * @param existing an open input stream obtained from this filesystem
+ * @return a new independent input stream for the same file
+ * @throws IOException if the stream cannot be cloned
+ */
+ public FSDataInputStream cloneDataInputStream(FSDataInputStream existing)
+ throws IOException {
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ InputStream wrapped = existing.getWrappedStream();
+ DFSInputStream dfsis;
+ if (wrapped instanceof DFSInputStream) {
+ dfsis = (DFSInputStream) wrapped;
+ } else if (wrapped instanceof org.apache.hadoop.crypto.CryptoInputStream) {
Review Comment:
you could use the java17 instanceof + variable declaration here as it's worth the effort; intellij does the work for you
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestOpenFileWithLocatedBlocks.java:
##########
@@ -0,0 +1,475 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import static org.apache.hadoop.test.MetricsAsserts.assertCounter;
+import static org.apache.hadoop.test.MetricsAsserts.getMetrics;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.crypto.CryptoInputStream;
+import org.apache.hadoop.crypto.key.JavaKeyStoreProvider;
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hdfs.client.CreateEncryptionZoneFlag;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests that opening a file with pre-fetched block locations via the
+ * {@code openFile().withFileStatus()} builder API skips the NameNode RPC
+ * for block locations.
+ */
+public class TestOpenFileWithLocatedBlocks {
+ private static final String NN_METRICS = "NameNodeActivity";
+ private static final int BLOCK_SIZE = 1024;
+ private static final short REPLICATION = 1;
+ private static final String TEST_KEY = "test_key";
+ private static final EnumSet<CreateEncryptionZoneFlag> NO_TRASH =
+ EnumSet.of(CreateEncryptionZoneFlag.NO_TRASH);
+
+ private MiniDFSCluster cluster;
+ private DistributedFileSystem fs;
+ private Configuration conf;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ conf = new HdfsConfiguration();
+ conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCK_SIZE);
+ FileSystemTestHelper fsHelper = new FileSystemTestHelper();
+ File testRootDir = new File(fsHelper.getTestRootDir()).getAbsoluteFile();
+ conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH,
+ JavaKeyStoreProvider.SCHEME_NAME + "://file" +
+ new Path(testRootDir.toString(), "test.jks").toUri());
+ cluster = new MiniDFSCluster.Builder(conf)
+ .numDataNodes(1)
+ .build();
+ cluster.waitActive();
+ fs = cluster.getFileSystem();
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (cluster != null) {
+ cluster.shutdown();
+ cluster = null;
+ }
+ }
+
+ private Path createFile(String name, byte[] data) throws IOException {
Review Comment:
org.apache.hadoop.fs.contract.ContractTestUtils do this and the readfully for you
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestOpenFileWithLocatedBlocks.java:
##########
@@ -0,0 +1,475 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import static org.apache.hadoop.test.MetricsAsserts.assertCounter;
+import static org.apache.hadoop.test.MetricsAsserts.getMetrics;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.crypto.CryptoInputStream;
+import org.apache.hadoop.crypto.key.JavaKeyStoreProvider;
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hdfs.client.CreateEncryptionZoneFlag;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests that opening a file with pre-fetched block locations via the
+ * {@code openFile().withFileStatus()} builder API skips the NameNode RPC
+ * for block locations.
+ */
+public class TestOpenFileWithLocatedBlocks {
+ private static final String NN_METRICS = "NameNodeActivity";
+ private static final int BLOCK_SIZE = 1024;
+ private static final short REPLICATION = 1;
+ private static final String TEST_KEY = "test_key";
+ private static final EnumSet<CreateEncryptionZoneFlag> NO_TRASH =
+ EnumSet.of(CreateEncryptionZoneFlag.NO_TRASH);
+
+ private MiniDFSCluster cluster;
+ private DistributedFileSystem fs;
+ private Configuration conf;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ conf = new HdfsConfiguration();
+ conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCK_SIZE);
+ FileSystemTestHelper fsHelper = new FileSystemTestHelper();
+ File testRootDir = new File(fsHelper.getTestRootDir()).getAbsoluteFile();
+ conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH,
+ JavaKeyStoreProvider.SCHEME_NAME + "://file" +
+ new Path(testRootDir.toString(), "test.jks").toUri());
+ cluster = new MiniDFSCluster.Builder(conf)
Review Comment:
hdfs tests must do it per test suite, so tests run faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4807573250
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 24s | | https://github.com/apache/hadoop/pull/8561 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/5/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#discussion_r3476293589
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestOpenFileWithLocatedBlocks.java:
##########
@@ -0,0 +1,475 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import static org.apache.hadoop.test.MetricsAsserts.assertCounter;
+import static org.apache.hadoop.test.MetricsAsserts.getMetrics;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
Review Comment:
copilot "unused imports are errors in Java" is utterly wrong there, obviously
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4801888077
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 48m 29s | | trunk passed |
| +1 :green_heart: | compile | 7m 29s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | compile | 7m 40s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | checkstyle | 3m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 38s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 6m 29s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 1m 51s | | branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 31m 52s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 40s | | the patch passed |
| +1 :green_heart: | compile | 5m 42s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | cc | 5m 42s | | the patch passed |
| +1 :green_heart: | golang | 5m 42s | | the patch passed |
| +1 :green_heart: | javac | 5m 42s | | the patch passed |
| +1 :green_heart: | compile | 6m 14s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | cc | 6m 14s | | the patch passed |
| +1 :green_heart: | golang | 6m 14s | | the patch passed |
| +1 :green_heart: | javac | 6m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 2m 42s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 0m 34s | | hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 28m 40s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 1s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 222m 4s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 10m 25s | | hadoop-hdfs-native-client in the patch passed. |
| +1 :green_heart: | asflicense | 1m 27s | | The patch does not generate ASF License warnings. |
| | | 444m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 499b3dbab600 5.15.0-177-generic #187-Ubuntu SMP Sat Apr 11 22:54:33 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 97f5a2df29c91cde9af42bbe4b654b274232bc05 |
| Default Java | Red Hat, Inc.-17.0.19+10-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.11.0.10-1.el8_10.x86_64:Red Hat, Inc.-21.0.11+10-LTS /usr/lib/jvm/java-17-openjdk-17.0.19.0.10-1.el8_10.x86_64:Red Hat, Inc.-17.0.19+10-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/testReport/ |
| Max. process+thread count | 4056 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/4/console |
| versions | git=2.43.7 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4814467568
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 50m 26s | | trunk passed |
| +1 :green_heart: | compile | 7m 20s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | compile | 7m 59s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | checkstyle | 3m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 55s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 6m 0s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 1m 37s | | branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 35m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 49s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 2s | | the patch passed |
| +1 :green_heart: | compile | 6m 7s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | cc | 6m 7s | | the patch passed |
| +1 :green_heart: | golang | 6m 7s | | the patch passed |
| +1 :green_heart: | javac | 6m 7s | | the patch passed |
| +1 :green_heart: | compile | 6m 35s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | cc | 6m 35s | | the patch passed |
| +1 :green_heart: | golang | 6m 35s | | the patch passed |
| +1 :green_heart: | javac | 6m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 2m 49s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 0m 32s | | hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 31m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 2s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 222m 27s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 10m 27s | | hadoop-hdfs-native-client in the patch passed. |
| +1 :green_heart: | asflicense | 1m 25s | | The patch does not generate ASF License warnings. |
| | | 454m 46s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 868db52e6d3e 5.15.0-177-generic #187-Ubuntu SMP Sat Apr 11 22:54:33 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1ad4359f47ac47a2eb0417eb839ba7fc71de4bee |
| Default Java | Red Hat, Inc.-17.0.19+10-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.11.0.10-1.el8_10.x86_64:Red Hat, Inc.-21.0.11+10-LTS /usr/lib/jvm/java-17-openjdk-17.0.19.0.10-1.el8_10.x86_64:Red Hat, Inc.-17.0.19+10-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/6/testReport/ |
| Max. process+thread count | 3875 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/6/console |
| versions | git=2.43.7 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "boroknagyz (via GitHub)" <gi...@apache.org>.
boroknagyz commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4818250961
Thanks, @steveloughran! Opened #8570 for branch-3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#discussion_r3476329252
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestOpenFileWithLocatedBlocks.java:
##########
@@ -66,12 +63,13 @@ public class TestOpenFileWithLocatedBlocks {
private static final EnumSet<CreateEncryptionZoneFlag> NO_TRASH =
EnumSet.of(CreateEncryptionZoneFlag.NO_TRASH);
- private MiniDFSCluster cluster;
- private DistributedFileSystem fs;
- private Configuration conf;
+ private static MiniDFSCluster cluster;
+ private static DistributedFileSystem fs;
+ private static Configuration conf;
+ private static Path ezPath;
- @BeforeEach
- public void setUp() throws IOException {
+ @BeforeAll
+ public static void setUp() throws Exception {
Review Comment:
very minor, just call this "setupCluster" and "teardownCluster".
setUp/tearDown have were the per method calls in junit1, and it's easy for people who have read too many tests to get mixed up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#discussion_r3476392775
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##########
@@ -378,6 +383,85 @@ public FSDataInputStream open(PathHandle fd, int bufferSize)
return dfs.createWrappedInputStream(dfsis);
}
+ @Override
+ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
+ final Path path,
+ final OpenFileParameters parameters) throws IOException {
+ AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
+ parameters.getMandatoryKeys(),
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS,
+ "for " + path);
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ final Path absF = fixRelativePart(path);
+ return LambdaUtils.eval(new CompletableFuture<>(), () -> {
+ LocatedBlocks locatedBlocks =
+ getLocatedBlocksFromStatus(parameters.getStatus());
+ final DFSInputStream dfsis;
+ if (locatedBlocks != null) {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum, locatedBlocks);
+ } else {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum);
+ }
+ try {
+ return dfs.createWrappedInputStream(dfsis);
+ } catch (IOException e) {
+ dfsis.close();
+ throw e;
+ }
+ });
+ }
+
+ private static LocatedBlocks getLocatedBlocksFromStatus(FileStatus status) {
Review Comment:
add one more check here, filenames. From the spec.
```
The final `status.getPath().getName()` element of the supplied status MUST equal
the name value of the path supplied to the `openFile(path)` call.
Filesystems MUST NOT validate the rest of the path.
This is needed to support viewfs and other mount-point wrapper filesystems
where schemas and paths are different. These often create their own FileStatus results
```
not quite so critical given you are already gating on class, but you don't want to have the status of a different file passed in, do you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "boroknagyz (via GitHub)" <gi...@apache.org>.
boroknagyz commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4811874568
Sure, thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#discussion_r3455045489
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestOpenFileWithLocatedBlocks.java:
##########
@@ -0,0 +1,475 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs;
+
+import static org.apache.hadoop.test.MetricsAsserts.assertCounter;
+import static org.apache.hadoop.test.MetricsAsserts.getMetrics;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
Review Comment:
Remove unused static imports; they cause compilation to fail (unused imports are errors in Java).
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##########
@@ -378,6 +383,72 @@ public FSDataInputStream open(PathHandle fd, int bufferSize)
return dfs.createWrappedInputStream(dfsis);
}
+ @Override
+ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
+ final Path path,
+ final OpenFileParameters parameters) throws IOException {
+ AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
+ parameters.getMandatoryKeys(),
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS,
+ "for " + path);
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ final Path absF = fixRelativePart(path);
+ return LambdaUtils.eval(new CompletableFuture<>(), () -> {
+ LocatedBlocks locatedBlocks =
+ getLocatedBlocksFromStatus(parameters.getStatus());
+ final DFSInputStream dfsis;
+ if (locatedBlocks != null) {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum, locatedBlocks);
+ } else {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum);
+ }
+ return dfs.createWrappedInputStream(dfsis);
+ });
Review Comment:
openFileWithOptions() currently bypasses DistributedFileSystem#open(Path,int), so symlink resolution via FileSystemLinkResolver is skipped. Also, if dfs.createWrappedInputStream(dfsis) throws, dfsis is leaked because it isn't closed in this method (unlike open(Path,int) which closes on wrap failure).
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##########
@@ -378,6 +383,72 @@ public FSDataInputStream open(PathHandle fd, int bufferSize)
return dfs.createWrappedInputStream(dfsis);
}
+ @Override
+ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
+ final Path path,
+ final OpenFileParameters parameters) throws IOException {
+ AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
+ parameters.getMandatoryKeys(),
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS,
+ "for " + path);
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ final Path absF = fixRelativePart(path);
+ return LambdaUtils.eval(new CompletableFuture<>(), () -> {
+ LocatedBlocks locatedBlocks =
+ getLocatedBlocksFromStatus(parameters.getStatus());
+ final DFSInputStream dfsis;
+ if (locatedBlocks != null) {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum, locatedBlocks);
+ } else {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum);
+ }
+ return dfs.createWrappedInputStream(dfsis);
+ });
+ }
+
+ private static LocatedBlocks getLocatedBlocksFromStatus(FileStatus status) {
+ if (status instanceof HdfsLocatedFileStatus) {
+ return ((HdfsLocatedFileStatus) status).getLocatedBlocks();
+ }
+ return null;
+ }
+
+ /**
+ * Create a new input stream for the same file as an existing stream,
+ * reusing its cached block locations to avoid a NameNode RPC.
+ * The returned stream is independent (its own position, buffers, etc.)
+ * but shares the same block location metadata.
+ *
+ * @param existing an open input stream obtained from this filesystem
+ * @return a new independent input stream for the same file
+ * @throws IOException if the stream cannot be cloned
+ */
+ public FSDataInputStream cloneDataInputStream(FSDataInputStream existing)
+ throws IOException {
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ InputStream wrapped = existing.getWrappedStream();
+ DFSInputStream dfsis;
+ if (wrapped instanceof DFSInputStream) {
+ dfsis = (DFSInputStream) wrapped;
+ } else if (wrapped instanceof org.apache.hadoop.crypto.CryptoInputStream) {
+ dfsis = (DFSInputStream)
+ ((org.apache.hadoop.crypto.CryptoInputStream) wrapped)
+ .getWrappedStream();
+ } else {
+ throw new IOException("Cannot clone: underlying stream is "
+ + wrapped.getClass().getName() + ", not a DFSInputStream");
+ }
+ LocatedBlocks locatedBlocks = dfsis.getLocatedBlocks();
+ String src = dfsis.getSrc();
+ DFSInputStream clone = dfs.open(src,
+ dfs.getConf().getIoBufferSize(), verifyChecksum, locatedBlocks);
+ return dfs.createWrappedInputStream(clone);
Review Comment:
cloneDataInputStream() should (1) verify the stream was opened by this DistributedFileSystem/DFSClient (to avoid cloning a stream from a different cluster instance) and (2) close the newly opened DFSInputStream if wrapping it (e.g., CryptoInputStream) fails, matching the error-handling pattern in open(Path,int).
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:
##########
@@ -1078,6 +1078,24 @@ public DFSInputStream open(String src, int buffersize, boolean verifyChecksum)
}
}
+ /**
+ * Create an input stream using pre-fetched block locations, skipping the
+ * NameNode RPC to get block locations.
+ * @param src file name
+ * @param buffersize ignored
+ * @param verifyChecksum verify checksums before returning data to client
+ * @param locatedBlocks pre-fetched block locations for this file
+ * @return an input stream for reading the file
+ * @throws IOException on I/O error
+ */
+ public DFSInputStream open(String src, int buffersize, boolean verifyChecksum,
+ LocatedBlocks locatedBlocks) throws IOException {
+ checkOpen();
+ try (TraceScope ignored = newPathTraceScope("newDFSInputStream", src)) {
+ return openInternal(locatedBlocks, src, verifyChecksum);
+ }
+ }
Review Comment:
The new DFSClient#open(String,int,boolean,LocatedBlocks) overload will currently throw a generic IOException("Cannot open filename ...") if locatedBlocks is null (via openInternal). This is a confusing failure mode for a caller; explicitly validate the parameter and fail fast with IllegalArgumentException.
##########
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include/hdfs/hdfs.h:
##########
@@ -685,11 +685,27 @@ extern "C" {
int hdfsCloseFile(hdfsFS fs, hdfsFile file);
- /**
- * hdfsExists - Checks if a given path exsits on the filesystem
+ /**
+ * hdfsCloneFile - Create a new independent file handle for the same file
+ * as an existing handle, reusing cached block locations to avoid a
+ * NameNode RPC. The cloned handle has its own read position and buffers.
+ * Only works for input (read) streams on a DistributedFileSystem.
+ *
+ * @param fs The configured filesystem handle.
+ * @param file An open input file handle to clone.
+ * @return Returns a new file handle on success, NULL on error.
+ * On error, errno will be set appropriately.
+ * The returned handle must be closed with hdfsCloseFile.
+ */
+ LIBHDFS_EXTERNAL
+ hdfsFile hdfsCloneFile(hdfsFS fs, hdfsFile file);
+
+
+ /**
+ * hdfsExists - Checks if a given path exsits on the filesystem
Review Comment:
Typo in comment: "exsits" → "exists".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "boroknagyz (via GitHub)" <gi...@apache.org>.
boroknagyz commented on code in PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#discussion_r3480020313
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##########
@@ -378,6 +383,85 @@ public FSDataInputStream open(PathHandle fd, int bufferSize)
return dfs.createWrappedInputStream(dfsis);
}
+ @Override
+ protected CompletableFuture<FSDataInputStream> openFileWithOptions(
+ final Path path,
+ final OpenFileParameters parameters) throws IOException {
+ AbstractFSBuilderImpl.rejectUnknownMandatoryKeys(
+ parameters.getMandatoryKeys(),
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_STANDARD_OPTIONS,
+ "for " + path);
+ statistics.incrementReadOps(1);
+ storageStatistics.incrementOpCounter(OpType.OPEN);
+ final Path absF = fixRelativePart(path);
+ return LambdaUtils.eval(new CompletableFuture<>(), () -> {
+ LocatedBlocks locatedBlocks =
+ getLocatedBlocksFromStatus(parameters.getStatus());
+ final DFSInputStream dfsis;
+ if (locatedBlocks != null) {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum, locatedBlocks);
+ } else {
+ dfsis = dfs.open(getPathName(absF), parameters.getBufferSize(),
+ verifyChecksum);
+ }
+ try {
+ return dfs.createWrappedInputStream(dfsis);
+ } catch (IOException e) {
+ dfsis.close();
+ throw e;
+ }
+ });
+ }
+
+ private static LocatedBlocks getLocatedBlocksFromStatus(FileStatus status) {
Review Comment:
Thanks, added the check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4816739751
merged! nice piece of work!
@boroknagyz - do a cherrypick for branch-3.5, submit as a new PR, cross ref from here and if Yetus is happy I will merge without any more review. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4783237366
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 8m 16s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 18s | | trunk passed |
| +1 :green_heart: | compile | 3m 57s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | compile | 4m 15s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 25s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 4m 21s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 1m 15s | | branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 18m 22s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 38s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 45s | | the patch passed |
| +1 :green_heart: | compile | 3m 2s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | cc | 3m 2s | | the patch passed |
| +1 :green_heart: | golang | 3m 2s | | the patch passed |
| +1 :green_heart: | javac | 3m 2s | | the patch passed |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | cc | 3m 18s | | the patch passed |
| +1 :green_heart: | golang | 3m 18s | | the patch passed |
| +1 :green_heart: | javac | 3m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 10s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 0m 24s | | hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 15m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 14s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 184m 1s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 6m 46s | | hadoop-hdfs-native-client in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not generate ASF License warnings. |
| | | 321m 41s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux ad7ab47e83de 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6af528b71345fd469d5c2439fd1a925e873873cd |
| Default Java | Red Hat, Inc.-17.0.19+10-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.11.0.10-1.el8_10.x86_64:Red Hat, Inc.-21.0.11+10-LTS /usr/lib/jvm/java-17-openjdk-17.0.19.0.10-1.el8_10.x86_64:Red Hat, Inc.-17.0.19+10-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/3/testReport/ |
| Max. process+thread count | 3872 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/3/console |
| versions | git=2.43.7 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17593. Allow setting block locations when opening streams [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8561:
URL: https://github.com/apache/hadoop/pull/8561#issuecomment-4781937862
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 47m 14s | | trunk passed |
| +1 :green_heart: | compile | 7m 25s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | compile | 7m 45s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | checkstyle | 3m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 8s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 6m 7s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 1m 42s | | branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 1s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 3s | | the patch passed |
| +1 :green_heart: | compile | 5m 49s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | cc | 5m 49s | | the patch passed |
| +1 :green_heart: | golang | 5m 49s | | the patch passed |
| +1 :green_heart: | javac | 5m 49s | | the patch passed |
| +1 :green_heart: | compile | 6m 8s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | cc | 6m 8s | | the patch passed |
| +1 :green_heart: | golang | 6m 8s | | the patch passed |
| +1 :green_heart: | javac | 6m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 2m 38s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +0 :ok: | spotbugs | 0m 35s | | hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 28m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 222m 51s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 10m 27s | | hadoop-hdfs-native-client in the patch passed. |
| +1 :green_heart: | asflicense | 1m 22s | | The patch does not generate ASF License warnings. |
| | | 445m 22s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8561 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 96dfc956859d 5.15.0-177-generic #187-Ubuntu SMP Sat Apr 11 22:54:33 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c515dc94ecdfbfb46b055d7b00e61858df3909a4 |
| Default Java | Red Hat, Inc.-17.0.19+10-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.11.0.10-1.el8_10.x86_64:Red Hat, Inc.-21.0.11+10-LTS /usr/lib/jvm/java-17-openjdk-17.0.19.0.10-1.el8_10.x86_64:Red Hat, Inc.-17.0.19+10-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/2/testReport/ |
| Max. process+thread count | 3932 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8561/2/console |
| versions | git=2.43.7 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org