[PR] Add use_fabric_endpoint parameter to MicrosoftAzure class [datafusion-python]

["djouallah (via GitHub)" <gi...@apache.org>]

djouallah opened a new pull request, #1357:
URL: https://github.com/apache/datafusion-python/pull/1357

   ## Summary
   
   This PR adds support for the `use_fabric_endpoint` parameter to the `MicrosoftAzure` object store class, enabling connections to Microsoft Fabric OneLake storage.
   
   Fixes #1356
   
   ## Problem
   
   The Python `MicrosoftAzure` class was missing the `use_fabric_endpoint` parameter that exists in the underlying Rust `object_store` crate. This prevented users from connecting to Microsoft Fabric OneLake storage, which requires Data Lake Storage Gen2 endpoints instead of Azure Blob Storage endpoints.
   
   **Current behavior**: Defaults to `https://onelake.blob.core.windows.net/...`  
   **Required behavior**: Support `https://onelake.dfs.fabric.microsoft.com/...`
   
   ## Changes
   
   Added the `use_fabric_endpoint` parameter to the `MicrosoftAzure` class in `src/store.rs`:
   
   1. Added parameter to PyO3 signature macro (line 79)
   2. Added parameter to function signature as `Option<bool>` (line 92)
   3. Added conditional builder chain call to `with_use_fabric_endpoint()` (lines 130-132)
   
   This implementation follows the exact same pattern as existing boolean parameters (`use_emulator`, `allow_http`).
   
   ## Usage Example
   
   ```python
   from datafusion.object_store import MicrosoftAzure
   from datafusion import SessionContext
   import os
   
   # Create OneLake store with Fabric endpoint
   onelake_store = MicrosoftAzure(
       container_name="tpch",
       account='onelake',
       bearer_token=os.environ["AZURE_STORAGE_TOKEN"],
       use_fabric_endpoint=True  # NEW PARAMETER
   )
   
   # Register and use
   ctx = SessionContext()
   ctx.register_object_store("abfss://tpch@onelake.dfs.fabric.microsoft.com/storage.Lakehouse", onelake_store, None)
   ```
   
   ## Testing Note
   
   Comprehensive end-to-end testing of this feature requires:
   - An active Microsoft Fabric workspace
   - OneLake storage with data
   - Valid authentication tokens for OneLake access
   
   I have manually verified that:
   - The parameter is accepted without errors
   - The object store can be successfully registered with a SessionContext
   - Queries against OneLake storage work correctly with `use_fabric_endpoint=True`
   
   The code change is minimal and follows existing patterns exactly, making it low-risk for existing functionality.
   
   ## Checklist
   
   - [x] Follows existing code patterns
   - [x] Minimal, focused change
   - [x] Manually tested with OneLake
   - [x] No new dependencies required


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] Add use_fabric_endpoint parameter to MicrosoftAzure class [datafusion-python]

["timsaucer (via GitHub)" <gi...@apache.org>]

timsaucer merged PR #1357:
URL: https://github.com/apache/datafusion-python/pull/1357


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org