[jira] [Commented] (HADOOP-19946) ABFS: Enable data locality aware reads for Azure Blob Storage

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-19946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127772#comment-18127772 ] 

ASF GitHub Bot commented on HADOOP-19946:
-----------------------------------------

bhattmanish98 opened a new pull request, #8830:
URL: https://github.com/apache/hadoop/pull/8830

   ### Description of PR
   
   Introduces blob-layout-based data locality for the ABFS driver's Blob endpoint client, allowing `AbfsInputStream` reads to fetch data directly from the storage node hosting the relevant byte range rather than a generic endpoint. Adds the `BlobLayout` model, an XML parser for the layout response, a process-wide `BlobLayoutCache` with weight-based eviction and non-blocking async prefetching backed by a dedicated executor, and wiring in `AbfsBlobClient`/`AbfsInputStream` to look up and use endpoint-aware reads with correctness fallback (partial/corrupted/ evicted-cache scenarios) to the standard read path.
   
   The feature is gated behind the `fs.azure.enable.data.locality` configuration flag, which defaults to `false` (disabled), so existing behavior is unchanged unless explicitly opted in.
   
   ### How was this patch tested?
   
   Includes unit and integration tests covering `BlobLayout` parsing, caching, eviction (capacity/idle/weight-based), and `AbfsInputStream` read behavior across scenarios: data not returned, corrupted/partial responses, and cache eviction during read.
   
   Additionally manually validated end-to-end against a live Azure Storage account with the Blob endpoint and data locality enabled, confirming the blob-layout lookup, caching, and endpoint-aware read path behave asexpected.
   
   ### For code changes:
   
   - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [x] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? *Note: Automated CI testing doesn't cover all cases so manual testing with cloud storage is still required.*
   - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   ### AI Tooling
   
   Contains content generated by GitHub Copilot.
   - [x] The PR includes the phrase "Contains content generated by <tool>" where <tool> is the name of the AI tool used.
   - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html




> ABFS: Enable data locality aware reads for Azure Blob Storage
> -------------------------------------------------------------
>
>                 Key: HADOOP-19946
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19946
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/azure
>            Reporter: Manish Bhatt
>            Assignee: Manish Bhatt
>            Priority: Major
>
> Introduce data locality support in ABFS to improve read performance for Azure Blob Storage by routing read requests to the endpoint that hosts the required data range. The goal is to reduce latency, improve throughput, and make read operations more efficient for large files and distributed workloads. This work adds blob layout discovery, range-to-endpoint mapping, locality-aware range selection, and caching of layout metadata to avoid repeated lookups. It also enables segmented read-ahead aligned with locality boundaries and adds safeguards for concurrency, eviction, timeout handling, and failure propagation needed for reliable production-scale reads.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org