[jira] [Comment Edited] (SPARK-6301) Unable to load external jars while submitting Spark Job

["raju patel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360301#comment-14360301 ] 

raju patel edited comment on SPARK-6301 at 3/13/15 1:02 PM:
------------------------------------------------------------

I am attaching a sample program to demostrate what exactly I am trying to achieve :

Java Program HelloWorld.java:
package com.psl.si;

public class HelloWorld {

	public void printHello() {
		System.out.println("Hello World");
	}

	public int addNum(int n1,int n2){
		return n1+n2;
	}
}

Python Program helloworld.py :
#imported SparkConf and SparkContext
from pyspark import SparkConf,SparkContext
from jnius import autoclass

conf = SparkConf().setMaster("local").setAppName("hello world")
sc = SparkContext(conf = conf)

def mapFunction():
	def addOneToList(irdd):
		#Java Class loading using jnius autoclass method
		helloWorld = autoclass('com.psl.si.HelloWorld')
		hw = helloWorld()
		#Calling printHello() method from HelloWorld Java class
		hw.printHello()
		#Calling addNum() method from HelloWorld Java class
		sum=hw.addNum(4,6)
		print "sum=",sum

                #Adding 1 to existing list
		olist=[]
		olist.append(irdd+1)
		return olist
	
	data = [1,2,3,4,5]
        #rdd from list
	distData = sc.parallelize(data)
	
	x=distData.map(addOneToList)
	
#Calling mapFunction
mapFunction()


Created Jar of HelloWorld program which is inside com.psl.si package
 jar -cvf HelloWorld.jar com  (as HelloWorld.class is inside com.psl.si.HelloWorld)

Finally submitting Spark Job:
spark-submit --jars HelloWorld.jar helloworld.py

Error that I am getting:
 c = find_javaclass(clsname)
  File "jnius_export_func.pxi", line 23, in jnius.find_javaclass (jnius/jnius.c:12356)
JavaException: Class not found 'com/psl/si/HelloWorld'

It is not able to load the class from the jar. Please give your suggestions if I am doing anything wrong.




was (Author: rajupats91):
I am attaching a sample program that to demostrate what exactly I am trying to achieve :

Java Program HelloWorld.java:
package com.psl.si;

public class HelloWorld {

	public void printHello() {
		System.out.println("Hello World");
	}

	public int addNum(int n1,int n2){
		return n1+n2;
	}
}

Python Program helloworld.py :
#imported SparkConf and SparkContext
from pyspark import SparkConf,SparkContext
from jnius import autoclass

conf = SparkConf().setMaster("local").setAppName("hello world")
sc = SparkContext(conf = conf)

def mapFunction():
	def addOneToList(irdd):
		#Java Class loading using jnius autoclass method
		helloWorld = autoclass('com.psl.si.HelloWorld')
		hw = helloWorld()
		#Calling printHello() method from HelloWorld Java class
		hw.printHello()
		#Calling addNum() method from HelloWorld Java class
		sum=hw.addNum(4,6)
		print "sum=",sum

                #Adding 1 to existing list
		olist=[]
		olist.append(irdd+1)
		return olist
	
	data = [1,2,3,4,5]
        #rdd from list
	distData = sc.parallelize(data)
	
	x=distData.map(addOneToList)
	
#Calling mapFunction
mapFunction()


Created Jar of HelloWorld program which is inside com.psl.si package
 jar -cvf HelloWorld.jar com  (as HelloWorld.class is inside com.psl.si.HelloWorld)

Finally submitting Spark Job:
spark-submit --jars HelloWorld.jar helloworld.py

Error that I am getting:
 c = find_javaclass(clsname)
  File "jnius_export_func.pxi", line 23, in jnius.find_javaclass (jnius/jnius.c:12356)
JavaException: Class not found 'com/psl/si/HelloWorld'

It is not able to load the class from the jar. Please give your suggestions if I am doing anything wrong.



> Unable to load external jars while submitting Spark Job
> -------------------------------------------------------
>
>                 Key: SPARK-6301
>                 URL: https://issues.apache.org/jira/browse/SPARK-6301
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, Spark Submit
>    Affects Versions: 1.2.0
>            Reporter: raju patel
>
> We are using Jnius to call Java functions from Python. But when we are trying to submit the job using Spark,it is not able to load the java classes that are provided in the --jars option, although it is successfully able to load python class.
> The Error is like this :
>  c = find_javaclass(clsname)
>  File "jnius_export_func.pxi", line 23, in jnius.find_javaclass (jnius/jnius.c:12815)
> JavaException: Class not found



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org