[jira] [Created] (SPARK-16182) companion object's shutdownhook not being invoked

["Christian Chua (JIRA)" <ji...@apache.org>]

Christian Chua created SPARK-16182:
--------------------------------------

             Summary: companion object's shutdownhook not being invoked
                 Key: SPARK-16182
                 URL: https://issues.apache.org/jira/browse/SPARK-16182
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.6.1
         Environment: OSX El Capitan (java "1.8.0_65"), Oracle Linux 6 (java 1.8.0_92-b14)
            Reporter: Christian Chua
            Priority: Critical


Spark streaming documentation recommends application developers create static connection pools. To clean up this pool, we add a shutdown hook.

The problem is that in spark 1.6.1, the shutdown hook for an executor will be called only for the first submitted job.  (on the second and subsequent job submissions, the shutdown hook for the executor will NOT be invoked)

problem not seen when using java 1.7
problem not seen when using spark 1.6.0

looks like this bug is caused by this modification from 1.6.0 to 1.6.1:
https://issues.apache.org/jira/browse/SPARK-12486

steps to reproduce the problem :

1.) install spark 1.6.1
2.) submit this basic spark application

import org.apache.spark.{ SparkContext, SparkConf }
object MyPool {
    def printToFile( f : java.io.File )( op : java.io.PrintWriter => Unit ) {
        val p = new java.io.PrintWriter(f)
        try {
            op(p)
        }
        finally {
            p.close()
        }
    }
    def myfunc( ) = {
        "a"
    }
    def createEvidence( ) = {
        printToFile(new java.io.File("/var/tmp/evidence.txt")) { p =>
            p.println("the evidence")
        }
    }
    sys.addShutdownHook {
        createEvidence()
    }
}

object BasicSpark {
    def main( args : Array[String] ) = {
        val sparkConf = new SparkConf().setAppName("BasicPi")
        val sc = new SparkContext(sparkConf)
        sc.parallelize(1 to 2).foreach { i => println("f : " + MyPool.myfunc())
        }
        sc.stop()
    }
}


3.) you will see that /var/tmp/evidence.txt is created
4.) now delete this file 
5.) submit a second job
6.) you will see that /var/tmp/evidence.txt is no longer created on the second submission

7.) if you use java 7 or spark 1.6.0, the evidence file will be created on the second and subsequent submits




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org