[jira] [Created] (SPARK-14983) Getting CompileException when feed an UDF with an array and another paramter

["Sky Yin (JIRA)" <ji...@apache.org>]

Sky Yin created SPARK-14983:
-------------------------------

             Summary: Getting CompileException when feed an UDF with an array and another paramter
                 Key: SPARK-14983
                 URL: https://issues.apache.org/jira/browse/SPARK-14983
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Spark Core
    Affects Versions: 1.5.0
         Environment: Spark 1.5.0 on EMR
            Reporter: Sky Yin


I'm trying to apply an UDF to an array column (generated from the {{array}} funciton in SparkSQL) and another string column and got this error:

(some long source code before this...)
{code}
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:392)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:412)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:409)
	at org.spark-project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
	at org.spark-project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
	... 31 more
Caused by: org.codehaus.commons.compiler.CompileException: Line 1038, Column 28: Redefinition of local variable "values" 
	at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10174)
	at org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2554)
	at org.codehaus.janino.UnitCompiler.access$4300(UnitCompiler.java:185)
	at org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:2434)
	at org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:2508)
	at org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2437)
	at org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2395)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2250)
	at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:822)
	at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:794)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:507)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:658)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:662)
	at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:185)
	at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:350)
	at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1035)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354)
	at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:769)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:532)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:393)
	at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:185)
	at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:347)
	at org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1139)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354)
	at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:322)
	at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:383)
	at org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:315)
	at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:233)
	at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:192)
	at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:84)
	at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:77)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:387)
	... 35 more
{code}

I tried to create another UDF with an array as the only parameter and that worked well as the array got passed into the Python function as a list without any problem.

Also I tried to define an UDF with an array and a string as paramters in Scala and got the same {{org.codehaus.commons.compiler.CompileException: Line 1038, Column 28: Redefinition of local variable "values"}} error.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org