You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ko...@apache.org on 2023/01/17 07:47:50 UTC
[arrow] branch master updated: GH-31506: [Python] Address docstrings in Streams and File Access (Factory Functions) (#33609)
This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new ef98a9774d GH-31506: [Python] Address docstrings in Streams and File Access (Factory Functions) (#33609)
ef98a9774d is described below
commit ef98a9774dbac7705df5369f86948c3af367e73f
Author: Alenka Frim <Al...@users.noreply.github.com>
AuthorDate: Tue Jan 17 08:47:42 2023 +0100
GH-31506: [Python] Address docstrings in Streams and File Access (Factory Functions) (#33609)
# Which issue does this PR close?
Closes #31506
# Rationale for this change
Ensure docstrings for [Streams and File Access](https://arrow.apache.org/docs/python/api/files.html) - Factory Functions - have an Examples section:
# What changes are included in this PR?
Added docstring examples for:
- `pyarrow.input_stream`
- `pyarrow.output_stream`
- `pyarrow.memory_map`
- `pyarrow.create_memory_map`
Also pytest argument ` --disable-warnings` for `pytest-cython` is added, see https://github.com/lgpage/pytest-cython/issues/3:
```python
=============================== warnings summary ===============================
opt/conda/envs/arrow/lib/python3.9/site-packages/pytest_cython/plugin.py:57
/opt/conda/envs/arrow/lib/python3.9/site-packages/pytest_cython/plugin.py:57: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to DoctestModule is deprecated. Please use the (path: pathlib.Path) argument instead.
See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
return DoctestModule.from_parent(parent, fspath=path)
```
# Are these changes tested?
Yes, locally with `pytest --doctest-cython --disable-warnings pyarrow` and on the CI with `Python / AMD64 Conda Python 3.9 Sphinx & Numpydoc ` build.
# Are there any user-facing changes?
No.
* Closes: #31506
Authored-by: Alenka Frim <fr...@gmail.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
---
docker-compose.yml | 5 ++-
python/pyarrow/io.pxi | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 104 insertions(+), 1 deletion(-)
diff --git a/docker-compose.yml b/docker-compose.yml
index 3774d55868..d1a3529451 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1101,7 +1101,10 @@ services:
LANG: "C.UTF-8"
BUILD_DOCS_CPP: "ON"
BUILD_DOCS_PYTHON: "ON"
- PYTEST_ARGS: "--doctest-modules --doctest-cython"
+ # GH-31506/GH-33609: Remove --disable-warnings once
+ # https://github.com/lgpage/pytest-cython/issues/24 is resolved
+ # and a new version that includes the fix is released.
+ PYTEST_ARGS: "--doctest-modules --doctest-cython --disable-warnings"
volumes: *conda-volumes
command:
["/arrow/ci/scripts/cpp_build.sh /arrow /build &&
diff --git a/python/pyarrow/io.pxi b/python/pyarrow/io.pxi
index 21c17b4d36..a8eb5b983b 100644
--- a/python/pyarrow/io.pxi
+++ b/python/pyarrow/io.pxi
@@ -943,6 +943,20 @@ def memory_map(path, mode='r'):
Returns
-------
mmap : MemoryMappedFile
+
+ Examples
+ --------
+ Reading from a memory map without any memory allocation or copying:
+
+ >>> import pyarrow as pa
+ >>> with pa.output_stream('example_mmap.txt') as stream:
+ ... stream.write(b'Constructing a buffer referencing the mapped memory')
+ ...
+ 51
+ >>> with pa.memory_map('example_mmap.txt') as mmap:
+ ... mmap.read_at(6,45)
+ ...
+ b'memory'
"""
_check_is_file(path)
@@ -971,6 +985,18 @@ def create_memory_map(path, size):
Returns
-------
mmap : MemoryMappedFile
+
+ Examples
+ --------
+ Create a file with a memory map:
+
+ >>> import pyarrow as pa
+ >>> with pa.create_memory_map('example_mmap_create.dat', 27) as mmap:
+ ... mmap.write(b'Create a memory-mapped file')
+ ... mmap.read_at(10, 9)
+ ...
+ 27
+ b'memory-map'
"""
return MemoryMappedFile.create(path, size)
@@ -2218,6 +2244,40 @@ def input_stream(source, compression='detect', buffer_size=None):
buffer_size : int, default None
If None or 0, no buffering will happen. Otherwise the size of the
temporary read buffer.
+
+ Examples
+ --------
+ Create a readable BufferReader (NativeFile) from a Buffer or a memoryview object:
+
+ >>> import pyarrow as pa
+ >>> buf = memoryview(b"some data")
+ >>> with pa.input_stream(buf) as stream:
+ ... stream.read(4)
+ ...
+ b'some'
+
+ Create a readable OSFile (NativeFile) from a string or file path:
+
+ >>> import gzip
+ >>> with gzip.open('example.gz', 'wb') as f:
+ ... f.write(b'some data')
+ ...
+ 9
+ >>> with pa.input_stream('example.gz') as stream:
+ ... stream.read()
+ ...
+ b'some data'
+
+ Create a readable PythonFile (NativeFile) from a a Python file object:
+
+ >>> with open('example.txt', mode='w') as f:
+ ... f.write('some text')
+ ...
+ 9
+ >>> with pa.input_stream('example.txt') as stream:
+ ... stream.read(6)
+ ...
+ b'some t'
"""
cdef NativeFile stream
@@ -2270,6 +2330,46 @@ def output_stream(source, compression='detect', buffer_size=None):
buffer_size : int, default None
If None or 0, no buffering will happen. Otherwise the size of the
temporary write buffer.
+
+ Examples
+ --------
+ Create a writable NativeFile from a pyarrow Buffer:
+
+ >>> import pyarrow as pa
+ >>> data = b"buffer data"
+ >>> empty_obj = bytearray(11)
+ >>> buf = pa.py_buffer(empty_obj)
+ >>> with pa.output_stream(buf) as stream:
+ ... stream.write(data)
+ ...
+ 11
+ >>> with pa.input_stream(buf) as stream:
+ ... stream.read(6)
+ ...
+ b'buffer'
+
+ or from a memoryview object:
+
+ >>> buf = memoryview(empty_obj)
+ >>> with pa.output_stream(buf) as stream:
+ ... stream.write(data)
+ ...
+ 11
+ >>> with pa.input_stream(buf) as stream:
+ ... stream.read()
+ ...
+ b'buffer data'
+
+ Create a writable NativeFile from a string or file path:
+
+ >>> with pa.output_stream('example_second.txt') as stream:
+ ... stream.write(b'Write some data')
+ ...
+ 15
+ >>> with pa.input_stream('example_second.txt') as stream:
+ ... stream.read()
+ ...
+ b'Write some data'
"""
cdef NativeFile stream