You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by neeravsalaria <ne...@gmail.com> on 2018/05/04 09:01:52 UTC
SparkContext taking time after adding jars and asking yarn for
resources
In my production setup spark is always taking 40 seconds between these steps
like a fixed counter is set. In my local lab these steps take exact 1
second. I am not able to find the exact root cause of this behaviour. My
Spark application is running on Hortonworks platform in yarn client mode.
Can someone guide me what is happening between these steps
18/05/04 *07:56:45* INFO spark.SparkContext: Added JAR
file:/app/jobs/jobs.jar at spark://10.233.69.5:37668/jars/jobs.jar with
timestamp 1525420605369
18/05/04 *07:57:26* WARN shortcircuit.DomainSocketFactory: The short-circuit
local reads feature cannot be used because libhadoop cannot be loaded.
18/05/04 *07:58:12* INFO client.AHSProxy: Connecting to Application History
server at gsidev001-mgt-01.thales.fr/192.168.1.11:10200
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org