ClassNotFoundException: $line11.$read$ when loading an HDFS text file with SparkQL in spark-shell

[Svend <sv...@gmail.com>]

Hi all, 


I just installed a mesos 0.19 cluster. I am failing to execute basic SparkQL
operations on text files with Spark 1.0.1 with the spark-shell.  


I have one Mesos master without zookeeper and 4 mesos slaves. 
All nodes are running JDK 1.7.51 and Scala 2.10.4. 
The spark package is uploaded to hdfs and the user running the mesos slave
has permission to access to it. 
I am runnning HDFS from the latest CDH5. 
I tried both with the pre-built CDH5 spark package available from
http://spark.apache.org/downloads.html and by packaging spark with sbt
0.13.2, JDK 1.7.51 and scala 2.10.4 as explained here
http://mesosphere.io/learn/run-spark-on-mesos/


No matter what I try, when I execute the following code on the spark-shell : 



The job fails with the following error reported by the mesos slave nodes: 






Note that runnning a simple map+reduce job on the same hdfs files with the
same installation works fine:




The hdfs files contain just plain csv files: 




spark-env.sh look like this: 






Any help, comment or pointer would be greatly appreciated!

Thanks in advance


Svend







--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-line11-read-when-loading-an-HDFS-text-file-with-SparkQL-in-spark-shell-tp9954.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: ClassNotFoundException: $line11.$read$ when loading an HDFS text file with SparkQL in spark-shell

[Svend <sv...@gmail.com>]

As suggested, I opened this Jira: 

https://issues.apache.org/jira/browse/SPARK-2576

S



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-line11-read-when-loading-an-HDFS-text-file-with-SparkQL-in-spark-shell-tp9954p10155.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: ClassNotFoundException: $line11.$read$ when loading an HDFS text file with SparkQL in spark-shell

[Svend <sv...@gmail.com>]

Hi, 

Yes, the error still occurs when we replace the lambdas with named
functions: 



(same error traces as in previous posts)




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-line11-read-when-loading-an-HDFS-text-file-with-SparkQL-in-spark-shell-tp9954p10154.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: ClassNotFoundException: $line11.$read$ when loading an HDFS text file with SparkQL in spark-shell

[Michael Armbrust <mi...@databricks.com>]

Hmmmm, it could be some weirdness with classloaders / Mesos / spark sql?

I'm curious if you would hit an error if there were no lambda functions
involved.  Perhaps if you load the data using jsonFile or parquetFile.

Either way, I'd file a JIRA.  Thanks!
On Jul 16, 2014 6:48 PM, "Svend" <sv...@gmail.com> wrote:

> Hi Michael,
>
> Thanks for your reply. Yes, the reduce triggered the actual execution, I
> got
> a total length (totalLength: 95068762, for the record).
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-line11-read-when-loading-an-HDFS-text-file-with-SparkQL-in-spark-shell-tp9954p9984.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Re: ClassNotFoundException: $line11.$read$ when loading an HDFS text file with SparkQL in spark-shell

[Svend <sv...@gmail.com>]

Hi Michael, 

Thanks for your reply. Yes, the reduce triggered the actual execution, I got
a total length (totalLength: 95068762, for the record). 





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-line11-read-when-loading-an-HDFS-text-file-with-SparkQL-in-spark-shell-tp9954p9984.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: ClassNotFoundException: $line11.$read$ when loading an HDFS text file with SparkQL in spark-shell

[Michael Armbrust <mi...@databricks.com>]

Oh, I'm sorry... reduce is also an operation....


On Wed, Jul 16, 2014 at 3:37 PM, Michael Armbrust <mi...@databricks.com>
wrote:

>
> Note that runnning a simple map+reduce job on the same hdfs files with the
>> same installation works fine:
>>
>
> Did you call collect() on the totalLength?  Otherwise nothing has
> actually executed.
>

Re: ClassNotFoundException: $line11.$read$ when loading an HDFS text file with SparkQL in spark-shell

[Michael Armbrust <mi...@databricks.com>]

> Note that runnning a simple map+reduce job on the same hdfs files with the
> same installation works fine:
>

Did you call collect() on the totalLength?  Otherwise nothing has actually
executed.