You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/09/05 21:46:04 UTC
(spark) branch master updated: [SPARK-48493][PYTHON][FOLLOW-UP] Fix documentation for Arrow batch support in Python Data Source
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f7292cbb6a35 [SPARK-48493][PYTHON][FOLLOW-UP] Fix documentation for Arrow batch support in Python Data Source
f7292cbb6a35 is described below
commit f7292cbb6a357d10c01af00d9f1fb1885e4819b9
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Thu Sep 5 14:45:50 2024 -0700
[SPARK-48493][PYTHON][FOLLOW-UP] Fix documentation for Arrow batch support in Python Data Source
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/46826 that mainly fixes documentation. The real main thing is to match the length of the title at:
```diff
- Python Datasource Reader with direct Arrow Batch support for improved performance
- -----------------------------------------------------------------------------------------
+ Python Data Source Reader with direct Arrow Batch support for improved performance
+ ----------------------------------------------------------------------------------
```
### Why are the changes needed?
Otherwise, Sphinx documentation build complains with warnings.
### Does this PR introduce _any_ user-facing change?
It fixes some typos in user-facing documentation.
### How was this patch tested?
Manually built the documentation
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48001 from HyukjinKwon/SPARK-48493-followup.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
python/docs/source/user_guide/sql/python_data_source.rst | 12 ++++++------
python/pyspark/sql/datasource.py | 10 ++++++----
python/pyspark/sql/worker/plan_data_source_read.py | 11 ++++++-----
3 files changed, 18 insertions(+), 15 deletions(-)
diff --git a/python/docs/source/user_guide/sql/python_data_source.rst b/python/docs/source/user_guide/sql/python_data_source.rst
index 9d9af4725c02..832987d19e5a 100644
--- a/python/docs/source/user_guide/sql/python_data_source.rst
+++ b/python/docs/source/user_guide/sql/python_data_source.rst
@@ -453,17 +453,17 @@ We can also use the same data source in streaming reader and writer
query = spark.readStream.format("fake").load().writeStream.format("fake").start("/output_path")
-Python Datasource Reader with direct Arrow Batch support for improved performance
------------------------------------------------------------------------------------------
-The Python Datasource Reader now supports direct yielding of Arrow Batches, which can significantly improve data processing performance. By using the efficient Arrow format,
+Python Data Source Reader with direct Arrow Batch support for improved performance
+----------------------------------------------------------------------------------
+The Python Datasource Reader supports direct yielding of Arrow Batches, which can significantly improve data processing performance. By using the efficient Arrow format,
this feature avoids the overhead of traditional row-by-row data processing, resulting in performance improvements of up to one order of magnitude, especially with large datasets.
**Enabling Arrow Batch Support**:
-To enable this feature, configure your custom DataSource to yield Arrow batches by returning pyarrow.RecordBatch objects within the `read` method of your `DataSourceReader`
+To enable this feature, configure your custom DataSource to yield Arrow batches by returning `pyarrow.RecordBatch` objects within the `read` method of your `DataSourceReader`
(or `DataSourceStreamReader`) implementation. This method simplifies data handling and reduces the number of I/O operations, particularly beneficial for large-scale data processing tasks.
**Arrow Batch Example**:
-The following example demonstrates how to implement a basic data source using Arrow Batch support.
+The following example demonstrates how to implement a basic Data Source using Arrow Batch support.
.. code-block:: python
@@ -474,7 +474,7 @@ The following example demonstrates how to implement a basic data source using Ar
# Define the ArrowBatchDataSource
class ArrowBatchDataSource(DataSource):
"""
- A data source for testing Arrow Batch Serialization
+ A Data Source for testing Arrow Batch Serialization
"""
@classmethod
diff --git a/python/pyspark/sql/datasource.py b/python/pyspark/sql/datasource.py
index 63e2c3d0d0a2..a51c96a9d178 100644
--- a/python/pyspark/sql/datasource.py
+++ b/python/pyspark/sql/datasource.py
@@ -350,10 +350,11 @@ class DataSourceReader(ABC):
Returns
-------
- iterator of tuples or pyarrow RecordBatch
+ iterator of tuples or PyArrow's `RecordBatch`
An iterator of tuples or rows. Each tuple or row will be converted to a row
in the final DataFrame.
- It can also return an iterator of pyarrow RecordBatch if the data source supports it.
+ It can also return an iterator of PyArrow's `RecordBatch` if the data source
+ supports it.
Examples
--------
@@ -471,10 +472,11 @@ class DataSourceStreamReader(ABC):
Returns
-------
- iterator of tuples or pyarrow RecordBatch
+ iterator of tuples or PyArrow's `RecordBatch`
An iterator of tuples or rows. Each tuple or row will be converted to a row
in the final DataFrame.
- It can also return an iterator of pyarrow RecordBatch if the data source supports it.
+ It can also return an iterator of PyArrow's `RecordBatch` if the data source
+ supports it.
"""
raise PySparkNotImplementedError(
errorClass="NOT_IMPLEMENTED",
diff --git a/python/pyspark/sql/worker/plan_data_source_read.py b/python/pyspark/sql/worker/plan_data_source_read.py
index 605df6962486..2af25fb52f15 100644
--- a/python/pyspark/sql/worker/plan_data_source_read.py
+++ b/python/pyspark/sql/worker/plan_data_source_read.py
@@ -65,9 +65,10 @@ def records_to_arrow_batches(
data_source: DataSource,
) -> Iterable[pa.RecordBatch]:
"""
- First check if the iterator yields pyarrow record batched, if so, yield them directly.
- Otherwise, convert an iterator of Python tuples to an iterator of pyarrow record batches.
- For each Python tuple, check the types of each field and append it to the records batch.
+ First check if the iterator yields PyArrow's `pyarrow.RecordBatch`, if so, yield
+ them directly. Otherwise, convert an iterator of Python tuples to an iterator
+ of pyarrow record batches. For each Python tuple, check the types of each field
+ and append it to the records batch.
"""
pa_schema = to_arrow_schema(return_type)
@@ -139,7 +140,7 @@ def records_to_arrow_batches(
"type": type(result).__name__,
"name": data_source.name(),
"supported_types": "tuple, list, `pyspark.sql.types.Row`,"
- " pyarrow RecordBatch",
+ " `pyarrow.RecordBatch`",
},
)
@@ -181,7 +182,7 @@ def main(infile: IO, outfile: IO) -> None:
This process then creates a `DataSourceReader` instance by calling the `reader` method
on the `DataSource` instance. Then it calls the `partitions()` method of the reader and
- constructs a Python Arrow Batch with the data using the `read()` method of the reader.
+ constructs a PyArrow's `RecordBatch` with the data using the `read()` method of the reader.
The partition values and the Arrow Batch are then serialized and sent back to the JVM
via the socket.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org