[jira] [Commented] (HDFS-17593) Allow setting block locations when opening streams

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18090650#comment-18090650 ] 

ASF GitHub Bot commented on HDFS-17593:
---------------------------------------

boroknagyz opened a new pull request, #8561:
URL: https://github.com/apache/hadoop/pull/8561

   Adds support for opening HDFS files with pre-fetched block locations, eliminating redundant NameNode RPCs when callers already have cached block metadata.
   
   Changes:
   - DFSClient.open() overload accepting LocatedBlocks
   - DistributedFileSystem.openFileWithOptions() override that extracts LocatedBlocks from HdfsLocatedFileStatus passed via withFileStatus()
   - DistributedFileSystem.cloneDataInputStream() to create independent file handles reusing block locations from an existing stream
   - libhdfs hdfsCloneFile() C API for cloning file handles via JNI
   - Integration tests verifying no NameNode RPC on clone/open with status
   
   Contains content generated by Claude Opus 4.6
   
   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
   -->
   
   ### Description of PR
   Adds support for opening HDFS files with pre-fetched block locations, eliminating redundant NameNode RPCs when callers already have cached block metadata.
   
   ### How was this patch tested?
   Unit tests.
   
   ### For code changes:
   
   - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   ### AI Tooling
   
   If an AI tool was used:
   
   - [x] The PR includes the phrase "Contains content generated by <tool>"
         where <tool> is the name of the AI tool used.
   - [x] My use of AI contributions follows the ASF legal policy
         https://www.apache.org/legal/generative-tooling.html




> Allow setting block locations when opening streams
> --------------------------------------------------
>
>                 Key: HDFS-17593
>                 URL: https://issues.apache.org/jira/browse/HDFS-17593
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Csaba Ringhofer
>            Priority: Major
>
> The HDFS client seems to always get block locations from the namenode when opening a file:
> https://github.com/apache/hadoop/blob/4525c7e35ea22d7a6350b8af10eb8d2ff68376e7/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L1099
> This leads to unnecessary RPCs in Apache Impala when doing reads, as the block locations are cached globally and the executors already have a good guess about the block locations when opening a stream. Unless the cached block locations are stale ideally no RPC should be made to the namenode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org