You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Xiaoye Sun <su...@gmail.com> on 2016/10/27 18:05:27 UTC

large scheduler delay in OnlineLDAOptimizer, (MLlib and LDA)

Hi,

I am running some experiments with OnlineLDAOptimizer in Spark 1.6.1. My
Spark cluster has 30 machines.

However, I found that the Scheduler delay at job/stage "reduce at
LDAOptimizer.scala:452" is extremely large when the LDA model is large. The
delay could be tens of seconds.

Does anyone know the reason for that?

Best,
Xiaoye