You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/10/24 10:45:34 UTC
[spark] branch master updated: [SPARK-40897][DOCS] Add some PySpark APIs to References
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e2e449e83cd [SPARK-40897][DOCS] Add some PySpark APIs to References
e2e449e83cd is described below
commit e2e449e83cdc743e7701ed93eae782ef9042f2d1
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Oct 24 19:45:21 2022 +0900
[SPARK-40897][DOCS] Add some PySpark APIs to References
### What changes were proposed in this pull request?
add following missing APIs to references:
- StorageLevel.MEMORY_AND_DISK_DESER
- TaskContext.cpus
- BarrierTaskContext.cpus
### Why are the changes needed?
they were missing in Reference
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
manually check, for `BarrierTaskContext.cpus`
```
In [10]: from pyspark import BarrierTaskContext
In [11]: rdd = spark.sparkContext.parallelize([1])
In [12]: rdd.barrier().mapPartitions(lambda _: [BarrierTaskContext.get().cpus()]).collect()
Out[12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```
Closes #38373 from zhengruifeng/py_doc_missing.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/docs/source/reference/pyspark.rst | 3 +++
1 file changed, 3 insertions(+)
diff --git a/python/docs/source/reference/pyspark.rst b/python/docs/source/reference/pyspark.rst
index c3afae10ddb..ec3df071639 100644
--- a/python/docs/source/reference/pyspark.rst
+++ b/python/docs/source/reference/pyspark.rst
@@ -262,10 +262,12 @@ Management
StorageLevel.DISK_ONLY_3
StorageLevel.MEMORY_AND_DISK
StorageLevel.MEMORY_AND_DISK_2
+ StorageLevel.MEMORY_AND_DISK_DESER
StorageLevel.MEMORY_ONLY
StorageLevel.MEMORY_ONLY_2
StorageLevel.OFF_HEAP
TaskContext.attemptNumber
+ TaskContext.cpus
TaskContext.get
TaskContext.getLocalProperty
TaskContext.partitionId
@@ -277,6 +279,7 @@ Management
BarrierTaskContext.allGather
BarrierTaskContext.attemptNumber
BarrierTaskContext.barrier
+ BarrierTaskContext.cpus
BarrierTaskContext.get
BarrierTaskContext.getLocalProperty
BarrierTaskContext.getTaskInfos
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org