[jira] [Commented] (HADOOP-19941) ABFS: Support Photon (Apache Arrow) based ListBlobs on Blob endpoint with XML fallback

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096747#comment-18096747 ] 

ASF GitHub Bot commented on HADOOP-19941:
-----------------------------------------

bhattmanish98 opened a new pull request, #8611:
URL: https://github.com/apache/hadoop/pull/8611

   ### Description of PR
   
   Adds config-gated support for consuming **ListBlobs responses in the Apache Arrow (Photon) format** on the ABFS Blob endpoint, with automatic, transparent fallback to the existing XML path and **no public API changes**.
   
   JIRA: [HADOOP-19941](https://issues.apache.org/jira/browse/HADOOP-19941)
   
   ### Behaviour
   
   - New config `fs.azure.photon.enabled` (**default `false`**, opt-in). When enabled, ABFS advertises Arrow on ListBlobs via an `Accept` header (`application/vnd.apache.arrow.stream, application/xml`); the service may honour it or fall back to XML.
   - Response handling selects a parser by the response `Content-Type` — Arrow routes to the new Arrow parser, otherwise the existing XML SAX parser is used. Both produce the same `BlobListResultSchema`, so all downstream processing (rename-pending filtering, pagination, directory rectification) is unchanged.
   - Malformed Arrow responses surface as `AbfsDriverException` through the existing error-handling model rather than silently degrading.
   
   ### Implementation
   
   - **`ResponseParserFactory`** chooses between `XmlListBlobResponseParser` and `ArrowListBlobParser` behind the `ListBlobResponseParser` interface.
   - **`ArrowListBlobParser`** reaches full parity with the XML parser: blob user metadata (Metadata map column), `hdi_isfolder=true` directory markers (case-insensitive; empty `mkdir` directories), implicit directories (`BlobPrefix` / `ResourceType` `blobprefix`|`directory`), copy properties, native `TimeStampSec`/`UInt8` vectors, and continuation via schema `NextMarker` metadata.
   - Arrow/XML timestamps normalized to a single **RFC 1123 GMT** representation for identical `FileStatus` values; Arrow allocator memory limit is configurable.
   - **Interrupt-safe parsing**: reads the (already fully buffered) body through a non-interruptible channel instead of `ArrowStreamReader`'s interruptible NIO channel, avoiding `ClosedByInterruptException` on interrupted threads — matching the XML path.
   
   ### Telemetry
   
   Adds Photon metrics: request count, response (Arrow served) count, fallback (XML returned) count, parse-failure count, and an end-to-end listing-latency duration tracker.
   
   ### How was this patch tested?
   
   - **Unit tests**: parser selection, Arrow parsing (metadata/directory markers, blobprefix, native vector types, multi-page, special characters, allocator-limit, malformed streams, interrupt tolerance), request-header application, and metrics.
   - **Integration tests**: XML/Arrow parity, fallback, pagination, and metrics against a Blob-endpoint account, plus additions to the existing list-status ITest.
   - Documentation updated in `blobEndpoint.md`.
   
   ### For code changes:
   
   - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-19941. Your PR title ...')?
   - [x] Object storage: has the patch been tested against the target store (Azure Blob endpoint)?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF? (adds `org.apache.arrow:arrow-vector`, Apache-2.0)




> ABFS: Support Photon (Apache Arrow) based ListBlobs on Blob endpoint with XML fallback
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-19941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19941
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/azure
>            Reporter: Manish Bhatt
>            Assignee: Manish Bhatt
>            Priority: Major
>
> Add config-gated support (fs.azure.photon.enabled, default off) for consuming ListBlobs responses in the Apache Arrow (Photon) format on the ABFS Blob endpoint. When enabled, ABFS advertises Arrow via an Accept header; the response Content-Type selects an Arrow or the existing XML parser, both producing identical FileStatus results, so downstream behaviour is unchanged. Includes automatic, transparent fallback to XML, full parsing parity (metadata, directory markers, implicit directories, copy properties, native timestamp/length vectors), interrupt-safe Arrow parsing, and Photon telemetry (request, response, fallback, parse-failure counts and listing latency). No public API changes. Covered by unit and integration tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org