You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "Mevada, Vatsal" <Me...@sky.optymyze.com> on 2017/01/04 07:49:01 UTC

SparkAppHandle returns unknown state forever

Hi,

I am launching a spark Job from Java application using SparkLauncher. My code is as follows:

SparkAppHandle jobHandle;
try {
    jobHandle = new SparkLauncher()
            .setSparkHome("C:\\spark-2.0.0-bin-hadoop2.7")
            .setAppResource("hdfs://server/inputs/test.jar")
            .setMainClass("com.test.TestJob")
            .setMaster("spark://server:6066")
            .setVerbose(true)
            .setDeployMode("cluster")
            .addAppArgs("abc")
            .startApplication();

} catch (IOException e) {
    throw new RuntimeException(e);
}

while(!jobHandle.getState().isFinal());


I can see my job running on SparkUI and also it is finishing without any errors.

However my java application never terminates since jobHandle.getState() always returns UNKNOWN state. what am I missing here? My spark API version is 2.0.0. One more detail that might be relevant is that my launcher application is running on windows.

Regards,
Vatsal