You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ko...@apache.org on 2025/03/15 05:34:40 UTC
(arrow) branch main updated: GH-45758 [Python] Add AzureFileSystem documentation (#45759)
This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 69682cd597 GH-45758 [Python] Add AzureFileSystem documentation (#45759)
69682cd597 is described below
commit 69682cd5971b8f6652c5f8f243aeb15a386e8b91
Author: Mike Barry <ja...@users.noreply.github.com>
AuthorDate: Sat Mar 15 01:34:35 2025 -0400
GH-45758 [Python] Add AzureFileSystem documentation (#45759)
### Rationale for this change
Missing documentation on AzureFileSystem
### What changes are included in this PR?
Added ArrowFileSystem documentation
### Are these changes tested?
Not yet
### Are there any user-facing changes?
Documentation
* GitHub Issue: #45758
Lead-authored-by: Mike Barry <gi...@barry.io>
Co-authored-by: Mike Barry <ja...@users.noreply.github.com>
Co-authored-by: Bryce Mecum <pe...@gmail.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
---
docs/source/python/api/filesystems.rst | 1 +
docs/source/python/filesystems.rst | 41 ++++++++++++++++++++++++++++++++++
2 files changed, 42 insertions(+)
diff --git a/docs/source/python/api/filesystems.rst b/docs/source/python/api/filesystems.rst
index 4b416abd9e..4bf61e4197 100644
--- a/docs/source/python/api/filesystems.rst
+++ b/docs/source/python/api/filesystems.rst
@@ -43,6 +43,7 @@ Filesystem Implementations
GcsFileSystem
HadoopFileSystem
SubTreeFileSystem
+ AzureFileSystem
To define filesystems with behavior implemented in Python:
diff --git a/docs/source/python/filesystems.rst b/docs/source/python/filesystems.rst
index 23d10aaaad..aaf5dfe4dd 100644
--- a/docs/source/python/filesystems.rst
+++ b/docs/source/python/filesystems.rst
@@ -42,6 +42,7 @@ Pyarrow implements natively the following filesystem subclasses:
* :ref:`filesystem-s3` (:class:`S3FileSystem`)
* :ref:`filesystem-gcs` (:class:`GcsFileSystem`)
* :ref:`filesystem-hdfs` (:class:`HadoopFileSystem`)
+* :ref:`filesystem-azurefs` (:class:`AzureFileSystem`)
It is also possible to use your own fsspec-compliant filesystem with pyarrow functionalities as described in the section :ref:`filesystem-fsspec`.
@@ -295,6 +296,46 @@ some environment variables.
In contrast to the legacy HDFS filesystem with ``pa.hdfs.connect``, setting
``CLASSPATH`` is not optional (pyarrow will not attempt to infer it).
+.. _filesystem-azurefs:
+
+Azure Storage File System
+-------------------------
+
+PyArrow implements natively an Azure filesystem for Azure Blob Storage with or
+without heirarchical namespace enabled.
+
+The :class:`AzureFileSystem` constructor has several options to configure the
+Azure Blob Storage connection (e.g. account name, account key, SAS token, etc.).
+
+If neither ``account_key`` or ``sas_token`` is specified a `DefaultAzureCredential <https://github.com/Azure/azure-sdk-for-cpp/blob/main/sdk/identity/azure-identity/README.md#defaultazurecredential>`__
+is used for authentication. This means it will try several types of authentication
+and go with the first one that works. If any authentication parameters are provided when
+initialising the FileSystem, they will be used instead of the default credential.
+
+Example showing how you can read contents from an Azure Blob Storage account::
+
+ >>> from pyarrow import fs
+ >>> azure_fs = fs.AzureFileSystem(account_name='myaccount')
+
+ # List all contents in a container, recursively
+ >>> azure_fs.get_file_info(fs.FileSelector('my-container', recursive=True))
+ [<FileInfo for 'my-container/File1': type=FileType.File, size=10>,
+ <FileInfo for 'my-container/File2': type=FileType.File, size=20>,
+ <FileInfo for 'my-container/Dir1': type=FileType.Directory>,
+ <FileInfo for 'my-container/Dir1/File3': type=FileType.File, size=30>]
+
+ # Open a file for reading and download its contents
+ >>> f = azure_fs.open_input_stream('my-container/File1')
+ >>> f.readall()
+ b'some data'
+
+For more details on the parameters and usage, refer to the :class:`AzureFileSystem` class documentation.
+
+.. seealso::
+
+ See the `Azure SDK for C++ documentation <https://github.com/Azure/azure-sdk-for-cpp>`__
+ for more information on authentication and configuration options.
+
.. _filesystem-fsspec:
Using fsspec-compatible filesystems with Arrow