Error using SQLContext in spark

[rakesh sharma <ra...@hotmail.com>]

Error: application failed with exceptionjava.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.<init>(Lorg/apache/spark/api/java/JavaSparkContext;)V        at examples.PersonRecordReader.getPersonRecords(PersonRecordReader.java:35)        at examples.PersonRecordReader.main(PersonRecordReader.java:17)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:606)        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367)        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


Hi All
I am getting the above exception when I am using SQLContext in spark jobs.The error occurs only with the insertion of these statements. The rdd is fine and it prints all correctly.The error occurs when creating dataframes. I am using maven dependencies version 1.3.1
public static void getPersonRecords(String...args) {		SparkConf sparkConf = new SparkConf().setAppName("SQLContext");		JavaSparkContext javaSparkContext = new JavaSparkContext(sparkConf);		JavaRDD<String> lines = javaSparkContext.textFile(args[0], 1);		JavaRDD<Person> personRecords = lines.map(new Function<String, Person>() {
			public Person call(String line) throws Exception {				System.out.println(line);				String[] rec = line.split(",");				return new Person(Integer.parseInt(rec[1].trim()), rec[0]);			}		});		for(Person p : personRecords.collect())  {			System.out.println(p.getName());		}		SQLContext sqlContext = new SQLContext(javaSparkContext);		DataFrame dataFrame = sqlContext.createDataFrame(personRecords, Person.class);	}

Please help me stuck with this since morning
thanksrakesh 		 	   		  

Re: Error using SQLContext in spark

[Akhil Das <ak...@sigmoidanalytics.com>]

You need to add the spark-sql dependencies
<http://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10/1.3.1>
in your projects build file.

Thanks
Best Regards

On Wed, Sep 2, 2015 at 6:41 PM, rakesh sharma <ra...@hotmail.com>
wrote:

> Error: application failed with exception
> java.lang.NoSuchMethodError:
> org.apache.spark.sql.SQLContext.<init>(Lorg/apache/spark/api/java/JavaSparkContext;)V
>         at
> examples.PersonRecordReader.getPersonRecords(PersonRecordReader.java:35)
>         at examples.PersonRecordReader.main(PersonRecordReader.java:17)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>
> Hi All
>
> I am getting the above exception when I am using SQLContext in spark jobs.
> The error occurs only with the insertion of these statements. The rdd is
> fine and it prints all correctly.
> The error occurs when creating dataframes. I am using maven dependencies
> version 1.3.1
>
> public static void getPersonRecords(String...args) {
> SparkConf sparkConf = new SparkConf().setAppName("SQLContext");
> JavaSparkContext javaSparkContext = new JavaSparkContext(sparkConf);
> JavaRDD<String> lines = javaSparkContext.textFile(args[0], 1);
> JavaRDD<Person> personRecords = lines.map(new Function<String, Person>() {
>
> public Person call(String line) throws Exception {
> System.out.println(line);
> String[] rec = line.split(",");
> return new Person(Integer.parseInt(rec[1].trim()), rec[0]);
> }
> });
> for(Person p : personRecords.collect())  {
> System.out.println(p.getName());
> }
> SQLContext sqlContext = new SQLContext(javaSparkContext);
> DataFrame dataFrame = sqlContext.createDataFrame(personRecords,
> Person.class);
> }
>
>
> Please help me stuck with this since morning
>
> thanks
> rakesh
>