executor lost failure and update spark.executor.memory

[Peng Xia <sp...@gmail.com>]

Hi,

I am using the databricks spark cluster (AWS), and testing on my scala
experiment.
I have some issue when training on a 10 GB data with
LogisticRegressionWithLBFGS algorithm.
The code block where I met the issue is as follows:
import org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
val algorithm = new LogisticRegressionWithLBFGS()
algorithm.run(training_set)

First I got a lot executor lost failure and java out of memory issues, then
I repartitioned my training_set with more partitions and the out of memory
issues are gone, but Still get executor lost failure.

My cluster has 72 cores and 500GB ram in total.
Can any one give some idea on this?

And one more question is that when I create a notebook in databricks
cluster, the spark context instance is already initialized. But the
spark.executor.memory is set as only 23 GB while my work node has 50GB.
Does any one know how I can update it to 50GB?


Thanks,
Best,
Peng