Yarn Spark on EMR

[SURAJ SHETH <sh...@gmail.com>]

Hi,
Yarn UI on 18080 stops receiving updates Spark jobs/tasks immediately after
it starts. We see only one task completed in the UI while the other hasn't
got any resources while in reality, more than 5 tasks would have completed.
Hadoop - Amazon 2.6
Spark - 1.5

Thanks and Regards,
Suraj Sheth

RE: Yarn Spark on EMR

["Bozeman, Christopher" <bo...@amazon.com>]

Suraj,

Spark History server is running on 18080 (http://spark.apache.org/docs/latest/monitoring.html) which is not going to give you are real-time update on a running Spark application.   Given this is Spark on YARN, you will need to view the Spark UI from the Application Master URL which can be found from the YARN Resource Manager UI (master node:8088) and it would be best to use a SOCKS proxy in order nicely resolve the URLs.

Best regards,
Christopher


From: SURAJ SHETH [mailto:shethsh@gmail.com]
Sent: Sunday, November 15, 2015 8:19 AM
To: user@spark.apache.org
Subject: Yarn Spark on EMR

Hi,
Yarn UI on 18080 stops receiving updates Spark jobs/tasks immediately after it starts. We see only one task completed in the UI while the other hasn't got any resources while in reality, more than 5 tasks would have completed.
Hadoop - Amazon 2.6
Spark - 1.5

Thanks and Regards,
Suraj Sheth