[jira] [Commented] (SPARK-40238) support scaleUpFactor and initialNumPartition in pyspark rdd API

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-40238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688353#comment-17688353 ] 

Apache Spark commented on SPARK-40238:
--------------------------------------

User 'wayneguow' has created a pull request for this issue:
https://github.com/apache/spark/pull/40009

> support scaleUpFactor and initialNumPartition in pyspark rdd API
> ----------------------------------------------------------------
>
>                 Key: SPARK-40238
>                 URL: https://issues.apache.org/jira/browse/SPARK-40238
>             Project: Spark
>          Issue Type: Story
>          Components: PySpark
>    Affects Versions: 3.4.0
>            Reporter: Ziqi Liu
>            Priority: Major
>
> This is a followup on https://issues.apache.org/jira/browse/SPARK-40211
> `scaleUpFactor` and `initialNumPartition` config are not supported yet in pyspark rdd take API
>  
> (see [https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L2799)]
> basically it hardcoded `scaleUpFactor` as 1 and `initialNumPartition` as 4, therefore pyspark rdd take API is inconsistent with scala API.
>  
> Anyone familiar with pyspark can help support this (referring to [scala implementation|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1448])?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org