You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeremy Davis <je...@speakeasy.net> on 2018/05/31 05:11:15 UTC

Limit on the number of Jobs per Application


I have an application that does many thousand univariate GLM regressions that seems to break down after completing around 25K jobs.
Plenty of resources: disk, network, memory, CPU are free, but eventually it is only scheduling on a few threads (out of 400+ possible on the cluster)
No task failures or blacklisting has happened.
Also trying to bring up the Spark UI Jobs tab gets slower and slower to refresh (eventually it doesn’t return)




Spark 2.3.0 on EMR.


Example:
val features = getFeatures(dataFrame)
val responses = getResponses(dataFrame)

//make f/r pairs
val cross = features.flatMap(x => responses.map(y => (x, y))).toSeq.par

cross.map(c=>(c._1,c._2,glm(sparkSession,dataFrame,Array(c._1),c._2,true))).toList