You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by yael aharon <ya...@gmail.com> on 2015/10/09 00:03:10 UTC

Unexplained sleep time

Hello,
I am working on improving the performance of our Spark on Yarn applications.
Scanning through the logs I found the following lines:


[2015-10-07T16:25:17.245-04:00] [DataProcessing] [INFO] []
[org.apache.spark.Logging$class] [tid:main] [userID:yarn] Started
progress reporter thread - sleep time : 5000
[2015-10-07T16:25:22.262-04:00] [DataProcessing] [INFO] []
[org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl] [tid:Reporter]
[userID:yarn] Received new token for : hostname:8041


As the log says, the main thread sleeps for 5 seconds. Is there a way to
configure/eliminate this sleep?
thanks, Yael

Re: Unexplained sleep time

Posted by Shixiong Zhu <zs...@gmail.com>.

You don't need to care about this sleep. It runs in a separate thread and
usually won't affect the performance of your application.

Best Regards,
Shixiong Zhu

2015-10-09 6:03 GMT+08:00 yael aharon <ya...@gmail.com>:

> Hello,
> I am working on improving the performance of our Spark on Yarn
> applications.
> Scanning through the logs I found the following lines:
>
>
> [2015-10-07T16:25:17.245-04:00] [DataProcessing] [INFO] [] [org.apache.spark.Logging$class] [tid:main] [userID:yarn] Started progress reporter thread - sleep time : 5000
> [2015-10-07T16:25:22.262-04:00] [DataProcessing] [INFO] [] [org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl] [tid:Reporter] [userID:yarn] Received new token for : hostname:8041
>
>
> As the log says, the main thread sleeps for 5 seconds. Is there a way to
> configure/eliminate this sleep?
> thanks, Yael
>