[GitHub] [arrow] nealrichardson commented on a change in pull request #8065: ARROW-9858: [Python][Docs] Add user guide for filesystems interface

[GitBox <gi...@apache.org>]

nealrichardson commented on a change in pull request #8065:
URL: https://github.com/apache/arrow/pull/8065#discussion_r478656124



##########
File path: docs/source/python/filesystems.rst
##########
@@ -34,21 +34,87 @@ underlying storage, are automatically dereferenced.  Only basic
 :class:`metadata <FileInfo>` about file entries, such as the file size
 and modification time, is made available.
 
-Types
+The core interface is represented by the base class :class:`FileSystem`.
+Concrete subclasses are available for various kinds of storage, such as local
+filesystem access (:class:`LocalFileSystem`), HDFS (:class:`HadoopFileSystem`)
+and Amazon S3-compatible storage (:class:`S3FileSystem`).
+
+
+Usage
 -----
 
-The core interface is represented by the base class :class:`FileSystem`.
-Concrete subclasses are available for various kinds of storage:
-:class:`local filesystem access <LocalFileSystem>`,
-:class:`HDFS <HadoopFileSystem>` and
-:class:`Amazon S3-compatible storage <S3FileSystem>`.
+A FileSystem object can be created with one of the constuctors (and check the
+respective constructor for its options)::
+
+   >>> from pyarrow import fs
+   >>> local = fs.LocalFileSystem()
+
+or alternatively inferred from a URI::
+
+   >>> s3, path = fs.FileSystem.from_uri("s3://my-bucket")
+   >>> s3
+   <pyarrow._s3fs.S3FileSystem at 0x7f6760cbf4f0>
+   >>> path
+   'my-bucket'
+
+
+Reading and writing files
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Several of the IO-related functions in PyArrow accept either a URI (and infer
+the filesystem) or an explicit ``filesystem`` argument to specify the filesystem
+to read or write from. For example the :meth:`pyarrow.parquet.read_table`
+function can be used in the following ways::
+
+   # using a URI
+   pq.read_table("s3://my-bucket")
+   # using a path and filesystem
+   s3 = fs.S3FileSystem(..)
+   pq.read_table("my-bucket", filesystem=s3)
+
+The filesystem interface further allows to open files for reading (input) or
+writing (output) directly, which can be combined with functions that work with
+file-like objects. For example::
+
+   with local.open_output_stream("test.arrow") as file:
+      with pa.RecordBatchFileWriter(file, table.schema) as writer:
+         writer.write_table(table)
+
+
+Listing files
+~~~~~~~~~~~~~
+
+Inspecting the directories and files on a filesystem can be done with the
+:meth:`FileSystem.get_file_info` method. To list the the contents of a
+directory, use the :class`FileSelector` object to specify the selection::
+
+   >>> local.get_file_info(fs.FileSelector("dataset/", recursive=True))
+   [<FileInfo for 'dataset/part=B': type=FileType.Directory>,
+    <FileInfo for 'dataset/part=B/data0.parquet': type=FileType.File, size=1564>,
+    <FileInfo for 'dataset/part=A': type=FileType.Directory>,
+    <FileInfo for 'dataset/part=A/data0.parquet': type=FileType.File, size=1564>]
+
+This returns a list of :class:`FileInfo` objects, containing information about
+the type (file or directory), the size, the date last modified, etc.
+
+You can also get this information for a single explicit path (or list of
+paths)::
+
+   >>> local.get_file_info(['test.arrow'])[0]
+   <FileInfo for 'test.arrow': type=FileType.File, size=3250>
+
+   >>> local.get_file_info(['non_existent'])
+   [<FileInfo for 'non_existent': type=FileType.NotFound>]
 
-Example
--------
+S3
+--
 
-Assuming your S3 credentials are correctly configured (for example by setting
-the ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY`` environment variables),
-here is how you can read contents from a S3 bucket::
+The :class:`S3FileSystem` constructor has several options to configure the S3
+connection. In addition, it will also read configured S3 credentials (for
+example by setting the ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY``

Review comment:
       How does this work? I don't see these env vars in our code; is this an aws-sdk-cpp feature?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org