CrossValidation distribution - is it in the roadmap?

[Tomasz Dudek <me...@gmail.com>]

Hey,

is there a way to make the following code:

val paramGrid = new ParamGridBuilder().//omitted for brevity - lets say we
have hundreds of param combinations here

val cv = new
CrossValidator().setNumFolds(3).setEstimator(pipeline).setEstimatorParamMaps(paramGrid)

automatically distribute itself over all the executors? What I mean is
to simultaneously compute few(or hundreds of it) ML models, instead of
using all the computation power on just one model at time.

If not, is such behavior in the Spark's road map?

...if not, do you think a person without prior Spark development
experience(me) could do it? I'm using SparkML daily, since few months, at
work. How much time would it take, approximately?

Yours,
Tomasz

Re: CrossValidation distribution - is it in the roadmap?

[Nick Pentreath <ni...@gmail.com>]

Hi Tomasz

Parallel evaluation for CrossValidation and TrainValidationSplit was added
for Spark 2.3 in https://issues.apache.org/jira/browse/SPARK-19357


On Wed, 29 Nov 2017 at 16:31 Tomasz Dudek <me...@gmail.com>
wrote:

> Hey,
>
> is there a way to make the following code:
>
> val paramGrid = new ParamGridBuilder().//omitted for brevity - lets say we
> have hundreds of param combinations here
>
> val cv = new
> CrossValidator().setNumFolds(3).setEstimator(pipeline).setEstimatorParamMaps(paramGrid)
>
> automatically distribute itself over all the executors? What I mean is
> to simultaneously compute few(or hundreds of it) ML models, instead of
> using all the computation power on just one model at time.
>
> If not, is such behavior in the Spark's road map?
>
> ...if not, do you think a person without prior Spark development
> experience(me) could do it? I'm using SparkML daily, since few months, at
> work. How much time would it take, approximately?
>
> Yours,
> Tomasz
>
>
>