You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/12/31 01:26:21 UTC
[spark] branch master updated: [SPARK-41553][PS][PYTHON][CORE] Fix the documentation for `num_files`
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b9ff737e3d6 [SPARK-41553][PS][PYTHON][CORE] Fix the documentation for `num_files`
b9ff737e3d6 is described below
commit b9ff737e3d68bcc9e788185eab8e320c1742ca41
Author: Bjørn <bj...@gmail.com>
AuthorDate: Sat Dec 31 10:26:07 2022 +0900
[SPARK-41553][PS][PYTHON][CORE] Fix the documentation for `num_files`
### What changes were proposed in this pull request?
"num_files: the number of files to be written ..."
`files` is changed to `partitions`
"num_files: the number of partitions to be written ..."
And there is added a text "This is deprecated. Use `DataFrame.spark.repartition` instead."
With this PR users can find the right documentation before they get an error message that they should use something other than what they have used.
### Why are the changes needed?
`num_files` has been deprecated and might be removed in a future version. "
"Use `DataFrame.spark.repartition` instead.",
The num_files argument doesn't manage the number of files, but specifying the partition number. https://github.com/apache/spark/pull/33379
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA
Closes #39098 from bjornjorgensen/num_files-to-repartition.
Lead-authored-by: Bjørn <bj...@gmail.com>
Co-authored-by: Bjørn Jørgensen <47...@users.noreply.github.com>
Co-authored-by: Bjørn Jørgensen <bj...@gmail.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/pandas/generic.py | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/python/pyspark/pandas/generic.py b/python/pyspark/pandas/generic.py
index c2a32f37767..9f218ac9a9f 100644
--- a/python/pyspark/pandas/generic.py
+++ b/python/pyspark/pandas/generic.py
@@ -671,8 +671,9 @@ class Frame(object, metaclass=ABCMeta):
.. note:: pandas-on-Spark writes CSV files into the directory, `path`, and writes
multiple `part-...` files in the directory when `path` is specified.
- This behavior was inherited from Apache Spark. The number of files can
- be controlled by `num_files`.
+ This behavior was inherited from Apache Spark. The number of partitions can
+ be controlled by `num_files`. This is deprecated.
+ Use `DataFrame.spark.repartition` instead.
Parameters
----------
@@ -694,8 +695,8 @@ class Frame(object, metaclass=ABCMeta):
escapechar: str, default None
String of length 1. Character used to escape `sep` and `quotechar`
when appropriate.
- num_files: the number of files to be written in `path` directory when
- this is a path.
+ num_files: the number of partitions to be written in `path` directory when
+ this is a path. This is deprecated. Use `DataFrame.spark.repartition` instead.
mode: str
Python write mode, default 'w'.
@@ -900,8 +901,9 @@ class Frame(object, metaclass=ABCMeta):
.. note:: pandas-on-Spark writes JSON files into the directory, `path`, and writes
multiple `part-...` files in the directory when `path` is specified.
- This behavior was inherited from Apache Spark. The number of files can
- be controlled by `num_files`.
+ This behavior was inherited from Apache Spark. The number of partitions can
+ be controlled by `num_files`. This is deprecated.
+ Use `DataFrame.spark.repartition` instead.
.. note:: output JSON format is different from pandas'. It always uses `orient='records'`
for its output. This behavior might have to change soon.
@@ -927,8 +929,8 @@ class Frame(object, metaclass=ABCMeta):
A string representing the compression to use in the output file,
only used when the first argument is a filename. By default, the
compression is inferred from the filename.
- num_files: the number of files to be written in `path` directory when
- this is a path.
+ num_files: the number of partitions to be written in `path` directory when
+ this is a path. This is deprecated. Use `DataFrame.spark.repartition` instead.
mode: str
Python write mode, default 'w'.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org