You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Ravi (JIRA)" <ji...@apache.org> on 2018/09/02 15:02:00 UTC

[jira] [Created] (SPARK-25309) Sci-kit Learn like Auto Pipeline Parallelization in Spark

Ravi created SPARK-25309:
----------------------------

             Summary: Sci-kit Learn like Auto Pipeline Parallelization in Spark 
                 Key: SPARK-25309
                 URL: https://issues.apache.org/jira/browse/SPARK-25309
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 2.3.1
            Reporter: Ravi


SPARK-19357 and SPARK-21911 have helped parallelize Pipelines in Spark. However, instead of setting the parallelism Parameter in the CrossValidator it would be good to have something like njobs=-1 (like Scikit Learn) where the Pipleline DAG could be automatically parallelized and scheduled based on the resources allocated to the Spark Session instead of having the user pick the integer value for this parameter. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org