[I] MicrosoftAzure does not support OneLake/Fabric endpoints [datafusion-python]

["djouallah (via GitHub)" <gi...@apache.org>]

djouallah opened a new issue, #1356:
URL: https://github.com/apache/datafusion-python/issues/1356

   ## Summary
   The `MicrosoftAzure` class in `datafusion.object_store` does not expose the `use_fabric_endpoint` parameter that exists in the underlying Rust `object_store` crate, making it impossible to connect to Microsoft Fabric OneLake storage.
   
   ## Environment
   - **datafusion-python version**: (51)
   - **Operating System**: (linux)
   
   ## Current Behavior
   When attempting to connect to OneLake using the `MicrosoftAzure` class with bearer token authentication, the object store defaults to using the Azure Blob Storage endpoint (`blob.core.windows.net`) instead of the required Data Lake Storage Gen2/Fabric endpoint (`dfs.fabric.microsoft.com`).
   
   This results in authentication errors when trying to access OneLake paths:
   
   ```python
   from datafusion.object_store import MicrosoftAzure
   from datafusion import SessionContext
   import os
   
   ctx = SessionContext()
   onelake_store = MicrosoftAzure(
       container_name="delta_rs",
       account='onelake',
       bearer_token=os.environ["AZURE_STORAGE_TOKEN"]
   )
   
   ctx.register_object_store(
       "abfss://delta_rs@onelake.dfs.fabric.microsoft.com/",
       onelake_store,
       None
   )
   
   ctx.sql("""
       CREATE EXTERNAL TABLE test
       STORED AS CSV
       LOCATION 'abfss://delta_rs@onelake.dfs.fabric.microsoft.com/test.Lakehouse/Files/csv'
   """)
   ```
   
   **Error:**
   ```
   DataFusion error: ObjectStore(Unauthenticated { 
       path: "test.Lakehouse/Files/csv", 
       source: RetryError(..., 
           uri: Some(https://onelake.blob.core.windows.net/delta_rs/test.Lakehouse/Files/csv), 
           ...
       )
   })
   ```
   
   Notice the URI shows `blob.core.windows.net` instead of `dfs.fabric.microsoft.com`.
   
   ## Expected Behavior
   The `MicrosoftAzure` class should expose a `use_fabric_endpoint` parameter (or similar) to allow connections to Microsoft Fabric OneLake, which uses the Data Lake Storage Gen2 endpoint format.
   
   The underlying Rust `object_store` crate already has this functionality via `MicrosoftAzureBuilder::with_use_fabric_endpoint()`:
   https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_use_fabric_endpoint
   
   ## Proposed Solution
   Add a `use_fabric_endpoint` parameter to the Python `MicrosoftAzure` class constructor:
   
   ```python
   onelake_store = MicrosoftAzure(
       container_name="delta_rs",
       account='onelake',
       bearer_token=os.environ["AZURE_STORAGE_TOKEN"],
       use_fabric_endpoint=True  # <- New parameter
   )
   ```
   
   This should map to the Rust builder's `with_use_fabric_endpoint()` method.
   
   ## Additional Context
   Microsoft Fabric OneLake is becoming increasingly popular for data lakehouse scenarios. Adding this support would enable datafusion-python users to query OneLake data directly, similar to how they can query S3, GCS, and regular Azure Blob Storage.
   
   OneLake uses the `abfss://` scheme with the endpoint format:
   `abfss://<workspace>@<account>.dfs.fabric.microsoft.com/<item>/<path>`
   
   ## Related Documentation
   - [[OneLake Access API](https://learn.microsoft.com/en-us/fabric/onelake/onelake-access-api)](https://learn.microsoft.com/en-us/fabric/onelake/onelake-access-api)
   - [[Rust object_store MicrosoftAzureBuilder](https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html)](https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] MicrosoftAzure does not support OneLake/Fabric endpoints [datafusion-python]

["timsaucer (via GitHub)" <gi...@apache.org>]

timsaucer closed issue #1356: MicrosoftAzure does not support OneLake/Fabric endpoints
URL: https://github.com/apache/datafusion-python/issues/1356


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org