Spark SQL - Registerfunction throwing MissingRequirementError in JavaMirror with primordial classloader

[Sunita Arvind <su...@gmail.com>]

Hi All,

I am trying to use a function within spark sql which accepts 2 - 4
arguments. I was able to get through compilation errors however, I see the
attached runtime exception when trying from Spark SQL.
(refer attachment for the complete stacktrace- StackTraceFor_runTestInSQL)

The function itself works well when tried as a regular function.

Here is how I am trying it:
************************************************************************************************************************************************
//Also tried without the defaulted  fmt parameters in the definition. The
issue persists.

 def within10yrs(FromDT: String, ToDT: String, fmt1:
DateTimeFormatter=dateFormats.YYYYMMDDHHMISS,fmt2:
DateTimeFormatter=dateFormats.YYYYMMDD):Boolean={
  println("Compute Within 10 years only if date2 is greater than date1")
     val yrsBetn = Years.yearsBetween(toDateTime(FromDT,fmt1),
toDateTime(ToDT,fmt2)).getYears
 val in10 =yrsBetn match {
                 case x if(x > 0 && x <= 10) => true
                 case _ => false
      }

  println("FromDT =" + FromDT + "ToDT =" + ToDT + "within10years =" + in10
+ " actual number of years is " + yrsBetn)
  in10
}

************************************************************************************************************************************************
def runTestInSQL(sc:SparkContext): Unit  = {
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    sqlContext.setConf("spark.sql.parquet.binaryAsString", "true")
    import sqlContext._
//Tried both the below variants to register the function:
   //     sqlContext.registerFunction("within10yrs",
(date1:String,date2:String,
fmt1:DateTimeFormatter,fmt2:DateTimeFormatter)=>within10yrs(date1,date2,fmt1,fmt2))
          sqlContext.registerFunction("within10yrs",
(date1:String,date2:String)=>within10yrs(date1,date2))

  val query1 = """ select a.col1,a.col2,a.col3,b.col4,b.col5
                                  FROM a JOIN b on a.col1 = b.col1
                      WHERE within10yrs(b.col4,a.col3)
GROUP  BY a.col1,
          a.col2,
          a.col3,
          b.col4,
          b.col5
"""
// The table b in the query above is a nested query actually and the sql
works well without the "WHERE within10yrs(b.col4,a.col3)". Hence skipping
the details to keep the problem
description simple.
 val res1 = sqlContext.sql(query1)
res1.count() // Line number 70 as in the stack trace
}

Execution throws runtime exception:
----------------------------------------------------
//Stack trace available in the file named - StackTraceFor_runTestInSQL
************************************************************************************************************************************************
def testasStdAlone()={
      val date1 = "2005-07-18 00:00:00"
      val fmt1 =  dateFormats.YYYYMMDDHHMISS
      val date2 = "20150719"
      val fmt2 =  dateFormats.YYYYMMDD
      println("date1 is " + date1 + " format is " +
dateFormats.YYYYMMDDHHMISS )
      println("date2 is " + date2 + " format is " + dateFormats.YYYYMMDD )
      //println("Within 10 years is " + within10yrs(date1,fmt1,date2,fmt2))
      println("Within 10 years is " + within10yrs(date1,date2))
    }

   def main(x: Array[String]): Unit = {
        val returned = testasStdAlone()
          //testRollup()
        println(returned)
    }
Output as expected:
----------------------------
date1 is 2005-07-18 00:00:00 format is
org.joda.time.format.DateTimeFormatter@28d101f3
date2 is 20150719 format is org.joda.time.format.DateTimeFormatter@5e411af2
Within 10 years
FromDT =2005-07-18 00:00:00ToDT =20150719within10years =true actual number
of years is 10
Within 10 years is true
()


Appreciate any direction from the community.

regards
Sunita