You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jonathan Chayat <jo...@supersonicads.com> on 2014/04/26 19:47:34 UTC
Using Spark in IntelliJ Scala Console
Hi all,
TLDR: running spark locally through IntelliJ IDEA Scala Console results
in java.lang.ClassNotFoundException
Long version:
I'm an algorithms developer in SupersonicAds - an ad network. We are
building a major new big data project and we are now in the process of
selecting our tech stack & tools.
I'm new to Spark, but I'm very excited about it. It is my opinion that
Spark can be a great tool for us, and that we might be able to build most
of our toolchain on top of it.
We currently develop in Scala and we are using IntelliJ IDEA as our IDE (we
love it). One of the features I love about IDEA is the Scala Console which
lets me work interactively with all of my project's code available and all
of the IDE's features & convenience. That is as opposed to the Scala Shell
& Spark Shell which I dislike because it is based on JLine and doesn't
behave like a good shell would (I cant even Ctrl-c to abort a line without
crashing the whole thing). Of course, as an algo guy, having a good REPL is
crucial to me.
To get started, I added the following line to build.sbt:
> "org.apache.spark" %% "spark-core" % "0.9.1"
Then, added the following main class:
import org.apache.spark.SparkContext
> import org.apache.spark.SparkContext._
>
object Main extends App {
> val sc = new SparkContext("local", "myApp")
> val r = sc.parallelize(1 to 1000)
> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 == 1).first())
> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 == 1).count())
> }
Make, Run, Works perfectly.
Next, I try running the same in the scala console.
Bad news - the last line throws an exception:
> ERROR executor.Executor: Exception in task ID 0
> java.lang.ClassNotFoundException: $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
It is my guess that for some reason Spark is not able to find the anonymous
function (_ % 2 == 1). Please note I'm running locally so I did not provide
any jars. For some reason when using first() instead of count() it works.
Needless to say it also works in Spark Shell but as I stated, working with
it is not an option.
This issue brings much sadness to my heart, and I could not find a solution
on the mailing list archives or elsewhere. I am hoping someone here might
offer some help.
Thanks,
Jon
Re: Using Spark in IntelliJ Scala Console
Posted by Jonathan Chayat <jo...@supersonicads.com>.
In IntelliJ, nothing changed. In SBT console I got this error:
$sbt
> console
[info] Running org.apache.spark.repl.Main -usejavacp
14/04/27 08:29:44 INFO spark.HttpServer: Starting HTTP Server
14/04/27 08:29:44 INFO server.Server: jetty-7.6.8.v20121106
14/04/27 08:29:44 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:50966
Failed to initialize compiler: object scala.annotation.Annotation in
compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programatically, settings.usejavacp.value = true.
14/04/27 08:29:44 WARN repl.SparkILoop$SparkILoopInterpreter: Warning:
compiler accessed before init set up. Assuming no postInit code.
Failed to initialize compiler: object scala.annotation.Annotation in
compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programatically, settings.usejavacp.value = true.
[error] (run-main-0) java.lang.AssertionError: assertion failed: null
java.lang.AssertionError: assertion failed: null
at scala.Predef$.assert(Predef.scala:179)
at
org.apache.spark.repl.SparkIMain.initializeSynchronous(SparkIMain.scala:197)
at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:924)
at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:881)
at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:881)
at
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:881)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:973)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
[trace] Stack trace suppressed: run last compile:runMain for the full
output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:runMain for the full
output.
[error] (compile:runMain) Nonzero exit code: 1
[error] Total time: 1 s, completed Apr 27, 2014 8:29:44 AM
On Sun, Apr 27, 2014 at 1:42 AM, Michael Armbrust <mi...@databricks.com>wrote:
> You'll also need:
>
> libraryDependencies += "org.apache.spark" %% "spark-repl" % <spark version>
>
>
>
> On Sat, Apr 26, 2014 at 3:32 PM, Michael Armbrust <mi...@databricks.com>wrote:
>
>> This is a little bit of a hack, but might work for you. You'll need to
>> be on sbt 0.13.2.
>>
>> connectInput in run := true
>>
>> outputStrategy in run := Some (StdoutOutput)
>>
>> console := {
>> (runMain in Compile).toTask(" org.apache.spark.repl.Main
>> -usejavacp").value
>> }
>>
>>
>> On Sat, Apr 26, 2014 at 1:05 PM, Jonathan Chayat <
>> jonathan.c@supersonicads.com> wrote:
>>
>>> Hi Michael, thanks for your prompt reply.
>>>
>>> It seems like IntelliJ Scala Console actually runs the Scala REPL (they
>>> print the same stuff when starting up).
>>> It is probably the SBT console.
>>>
>>> When I tried the same code in the Scala REPL of my project using "sbt
>>> console" it didn't work either.
>>> It only worked in spark project's bin/spark-shell
>>>
>>> Is there a way to customize the SBT console of a project listing spark
>>> as a dependency?
>>>
>>> Thx,
>>> Jon
>>>
>>>
>>> On Sat, Apr 26, 2014 at 9:42 PM, Michael Armbrust <
>>> michael@databricks.com> wrote:
>>>
>>>> The spark-shell is a special version of the Scala REPL that serves the
>>>> classes created for each line over HTTP. Do you know if the InteliJ Spark
>>>> console is just the normal Scala repl in a GUI wrapper, or if it is
>>>> something else entirely? If its the former, perhaps it might be possible
>>>> to tell InteliJ to bring up the spark version instead.
>>>>
>>>>
>>>> On Sat, Apr 26, 2014 at 10:47 AM, Jonathan Chayat <
>>>> jonathan.c@supersonicads.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> TLDR: running spark locally through IntelliJ IDEA Scala Console
>>>>> results in java.lang.ClassNotFoundException
>>>>>
>>>>> Long version:
>>>>>
>>>>> I'm an algorithms developer in SupersonicAds - an ad network. We are
>>>>> building a major new big data project and we are now in the process of
>>>>> selecting our tech stack & tools.
>>>>>
>>>>> I'm new to Spark, but I'm very excited about it. It is my opinion that
>>>>> Spark can be a great tool for us, and that we might be able to build most
>>>>> of our toolchain on top of it.
>>>>>
>>>>> We currently develop in Scala and we are using IntelliJ IDEA as our
>>>>> IDE (we love it). One of the features I love about IDEA is the Scala
>>>>> Console which lets me work interactively with all of my project's code
>>>>> available and all of the IDE's features & convenience. That is as opposed
>>>>> to the Scala Shell & Spark Shell which I dislike because it is based on
>>>>> JLine and doesn't behave like a good shell would (I cant even Ctrl-c to
>>>>> abort a line without crashing the whole thing). Of course, as an algo guy,
>>>>> having a good REPL is crucial to me.
>>>>>
>>>>> To get started, I added the following line to build.sbt:
>>>>>
>>>>>> "org.apache.spark" %% "spark-core" % "0.9.1"
>>>>>
>>>>>
>>>>> Then, added the following main class:
>>>>>
>>>>> import org.apache.spark.SparkContext
>>>>>> import org.apache.spark.SparkContext._
>>>>>>
>>>>>
>>>>>
>>>>> object Main extends App {
>>>>>> val sc = new SparkContext("local", "myApp")
>>>>>> val r = sc.parallelize(1 to 1000)
>>>>>> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 ==
>>>>>> 1).first())
>>>>>> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 ==
>>>>>> 1).count())
>>>>>> }
>>>>>
>>>>>
>>>>> Make, Run, Works perfectly.
>>>>>
>>>>> Next, I try running the same in the scala console.
>>>>> Bad news - the last line throws an exception:
>>>>>
>>>>>> ERROR executor.Executor: Exception in task ID 0
>>>>>> java.lang.ClassNotFoundException:
>>>>>> $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
>>>>>
>>>>>
>>>>> It is my guess that for some reason Spark is not able to find the
>>>>> anonymous function (_ % 2 == 1). Please note I'm running locally so I did
>>>>> not provide any jars. For some reason when using first() instead of count()
>>>>> it works. Needless to say it also works in Spark Shell but as I stated,
>>>>> working with it is not an option.
>>>>>
>>>>> This issue brings much sadness to my heart, and I could not find a
>>>>> solution on the mailing list archives or elsewhere. I am hoping someone
>>>>> here might offer some help.
>>>>>
>>>>> Thanks,
>>>>> Jon
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Using Spark in IntelliJ Scala Console
Posted by Michael Armbrust <mi...@databricks.com>.
You'll also need:
libraryDependencies += "org.apache.spark" %% "spark-repl" % <spark version>
On Sat, Apr 26, 2014 at 3:32 PM, Michael Armbrust <mi...@databricks.com>wrote:
> This is a little bit of a hack, but might work for you. You'll need to be
> on sbt 0.13.2.
>
> connectInput in run := true
>
> outputStrategy in run := Some (StdoutOutput)
>
> console := {
> (runMain in Compile).toTask(" org.apache.spark.repl.Main
> -usejavacp").value
> }
>
>
> On Sat, Apr 26, 2014 at 1:05 PM, Jonathan Chayat <
> jonathan.c@supersonicads.com> wrote:
>
>> Hi Michael, thanks for your prompt reply.
>>
>> It seems like IntelliJ Scala Console actually runs the Scala REPL (they
>> print the same stuff when starting up).
>> It is probably the SBT console.
>>
>> When I tried the same code in the Scala REPL of my project using "sbt
>> console" it didn't work either.
>> It only worked in spark project's bin/spark-shell
>>
>> Is there a way to customize the SBT console of a project listing spark as
>> a dependency?
>>
>> Thx,
>> Jon
>>
>>
>> On Sat, Apr 26, 2014 at 9:42 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> The spark-shell is a special version of the Scala REPL that serves the
>>> classes created for each line over HTTP. Do you know if the InteliJ Spark
>>> console is just the normal Scala repl in a GUI wrapper, or if it is
>>> something else entirely? If its the former, perhaps it might be possible
>>> to tell InteliJ to bring up the spark version instead.
>>>
>>>
>>> On Sat, Apr 26, 2014 at 10:47 AM, Jonathan Chayat <
>>> jonathan.c@supersonicads.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> TLDR: running spark locally through IntelliJ IDEA Scala Console results
>>>> in java.lang.ClassNotFoundException
>>>>
>>>> Long version:
>>>>
>>>> I'm an algorithms developer in SupersonicAds - an ad network. We are
>>>> building a major new big data project and we are now in the process of
>>>> selecting our tech stack & tools.
>>>>
>>>> I'm new to Spark, but I'm very excited about it. It is my opinion that
>>>> Spark can be a great tool for us, and that we might be able to build most
>>>> of our toolchain on top of it.
>>>>
>>>> We currently develop in Scala and we are using IntelliJ IDEA as our IDE
>>>> (we love it). One of the features I love about IDEA is the Scala Console
>>>> which lets me work interactively with all of my project's code available
>>>> and all of the IDE's features & convenience. That is as opposed to the
>>>> Scala Shell & Spark Shell which I dislike because it is based on JLine and
>>>> doesn't behave like a good shell would (I cant even Ctrl-c to abort a line
>>>> without crashing the whole thing). Of course, as an algo guy, having a good
>>>> REPL is crucial to me.
>>>>
>>>> To get started, I added the following line to build.sbt:
>>>>
>>>>> "org.apache.spark" %% "spark-core" % "0.9.1"
>>>>
>>>>
>>>> Then, added the following main class:
>>>>
>>>> import org.apache.spark.SparkContext
>>>>> import org.apache.spark.SparkContext._
>>>>>
>>>>
>>>>
>>>> object Main extends App {
>>>>> val sc = new SparkContext("local", "myApp")
>>>>> val r = sc.parallelize(1 to 1000)
>>>>> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 ==
>>>>> 1).first())
>>>>> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 ==
>>>>> 1).count())
>>>>> }
>>>>
>>>>
>>>> Make, Run, Works perfectly.
>>>>
>>>> Next, I try running the same in the scala console.
>>>> Bad news - the last line throws an exception:
>>>>
>>>>> ERROR executor.Executor: Exception in task ID 0
>>>>> java.lang.ClassNotFoundException:
>>>>> $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
>>>>
>>>>
>>>> It is my guess that for some reason Spark is not able to find the
>>>> anonymous function (_ % 2 == 1). Please note I'm running locally so I did
>>>> not provide any jars. For some reason when using first() instead of count()
>>>> it works. Needless to say it also works in Spark Shell but as I stated,
>>>> working with it is not an option.
>>>>
>>>> This issue brings much sadness to my heart, and I could not find a
>>>> solution on the mailing list archives or elsewhere. I am hoping someone
>>>> here might offer some help.
>>>>
>>>> Thanks,
>>>> Jon
>>>>
>>>
>>>
>>
>
Re: Using Spark in IntelliJ Scala Console
Posted by Michael Armbrust <mi...@databricks.com>.
This is a little bit of a hack, but might work for you. You'll need to be
on sbt 0.13.2.
connectInput in run := true
outputStrategy in run := Some (StdoutOutput)
console := {
(runMain in Compile).toTask(" org.apache.spark.repl.Main
-usejavacp").value
}
On Sat, Apr 26, 2014 at 1:05 PM, Jonathan Chayat <
jonathan.c@supersonicads.com> wrote:
> Hi Michael, thanks for your prompt reply.
>
> It seems like IntelliJ Scala Console actually runs the Scala REPL (they
> print the same stuff when starting up).
> It is probably the SBT console.
>
> When I tried the same code in the Scala REPL of my project using "sbt
> console" it didn't work either.
> It only worked in spark project's bin/spark-shell
>
> Is there a way to customize the SBT console of a project listing spark as
> a dependency?
>
> Thx,
> Jon
>
>
> On Sat, Apr 26, 2014 at 9:42 PM, Michael Armbrust <mi...@databricks.com>wrote:
>
>> The spark-shell is a special version of the Scala REPL that serves the
>> classes created for each line over HTTP. Do you know if the InteliJ Spark
>> console is just the normal Scala repl in a GUI wrapper, or if it is
>> something else entirely? If its the former, perhaps it might be possible
>> to tell InteliJ to bring up the spark version instead.
>>
>>
>> On Sat, Apr 26, 2014 at 10:47 AM, Jonathan Chayat <
>> jonathan.c@supersonicads.com> wrote:
>>
>>> Hi all,
>>>
>>> TLDR: running spark locally through IntelliJ IDEA Scala Console results
>>> in java.lang.ClassNotFoundException
>>>
>>> Long version:
>>>
>>> I'm an algorithms developer in SupersonicAds - an ad network. We are
>>> building a major new big data project and we are now in the process of
>>> selecting our tech stack & tools.
>>>
>>> I'm new to Spark, but I'm very excited about it. It is my opinion that
>>> Spark can be a great tool for us, and that we might be able to build most
>>> of our toolchain on top of it.
>>>
>>> We currently develop in Scala and we are using IntelliJ IDEA as our IDE
>>> (we love it). One of the features I love about IDEA is the Scala Console
>>> which lets me work interactively with all of my project's code available
>>> and all of the IDE's features & convenience. That is as opposed to the
>>> Scala Shell & Spark Shell which I dislike because it is based on JLine and
>>> doesn't behave like a good shell would (I cant even Ctrl-c to abort a line
>>> without crashing the whole thing). Of course, as an algo guy, having a good
>>> REPL is crucial to me.
>>>
>>> To get started, I added the following line to build.sbt:
>>>
>>>> "org.apache.spark" %% "spark-core" % "0.9.1"
>>>
>>>
>>> Then, added the following main class:
>>>
>>> import org.apache.spark.SparkContext
>>>> import org.apache.spark.SparkContext._
>>>>
>>>
>>>
>>> object Main extends App {
>>>> val sc = new SparkContext("local", "myApp")
>>>> val r = sc.parallelize(1 to 1000)
>>>> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 ==
>>>> 1).first())
>>>> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 ==
>>>> 1).count())
>>>> }
>>>
>>>
>>> Make, Run, Works perfectly.
>>>
>>> Next, I try running the same in the scala console.
>>> Bad news - the last line throws an exception:
>>>
>>>> ERROR executor.Executor: Exception in task ID 0
>>>> java.lang.ClassNotFoundException:
>>>> $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
>>>
>>>
>>> It is my guess that for some reason Spark is not able to find the
>>> anonymous function (_ % 2 == 1). Please note I'm running locally so I did
>>> not provide any jars. For some reason when using first() instead of count()
>>> it works. Needless to say it also works in Spark Shell but as I stated,
>>> working with it is not an option.
>>>
>>> This issue brings much sadness to my heart, and I could not find a
>>> solution on the mailing list archives or elsewhere. I am hoping someone
>>> here might offer some help.
>>>
>>> Thanks,
>>> Jon
>>>
>>
>>
>
Re: Using Spark in IntelliJ Scala Console
Posted by Jonathan Chayat <jo...@supersonicads.com>.
Hi Michael, thanks for your prompt reply.
It seems like IntelliJ Scala Console actually runs the Scala REPL (they
print the same stuff when starting up).
It is probably the SBT console.
When I tried the same code in the Scala REPL of my project using "sbt
console" it didn't work either.
It only worked in spark project's bin/spark-shell
Is there a way to customize the SBT console of a project listing spark as a
dependency?
Thx,
Jon
On Sat, Apr 26, 2014 at 9:42 PM, Michael Armbrust <mi...@databricks.com>wrote:
> The spark-shell is a special version of the Scala REPL that serves the
> classes created for each line over HTTP. Do you know if the InteliJ Spark
> console is just the normal Scala repl in a GUI wrapper, or if it is
> something else entirely? If its the former, perhaps it might be possible
> to tell InteliJ to bring up the spark version instead.
>
>
> On Sat, Apr 26, 2014 at 10:47 AM, Jonathan Chayat <
> jonathan.c@supersonicads.com> wrote:
>
>> Hi all,
>>
>> TLDR: running spark locally through IntelliJ IDEA Scala Console results
>> in java.lang.ClassNotFoundException
>>
>> Long version:
>>
>> I'm an algorithms developer in SupersonicAds - an ad network. We are
>> building a major new big data project and we are now in the process of
>> selecting our tech stack & tools.
>>
>> I'm new to Spark, but I'm very excited about it. It is my opinion that
>> Spark can be a great tool for us, and that we might be able to build most
>> of our toolchain on top of it.
>>
>> We currently develop in Scala and we are using IntelliJ IDEA as our IDE
>> (we love it). One of the features I love about IDEA is the Scala Console
>> which lets me work interactively with all of my project's code available
>> and all of the IDE's features & convenience. That is as opposed to the
>> Scala Shell & Spark Shell which I dislike because it is based on JLine and
>> doesn't behave like a good shell would (I cant even Ctrl-c to abort a line
>> without crashing the whole thing). Of course, as an algo guy, having a good
>> REPL is crucial to me.
>>
>> To get started, I added the following line to build.sbt:
>>
>>> "org.apache.spark" %% "spark-core" % "0.9.1"
>>
>>
>> Then, added the following main class:
>>
>> import org.apache.spark.SparkContext
>>> import org.apache.spark.SparkContext._
>>>
>>
>>
>> object Main extends App {
>>> val sc = new SparkContext("local", "myApp")
>>> val r = sc.parallelize(1 to 1000)
>>> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 ==
>>> 1).first())
>>> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 ==
>>> 1).count())
>>> }
>>
>>
>> Make, Run, Works perfectly.
>>
>> Next, I try running the same in the scala console.
>> Bad news - the last line throws an exception:
>>
>>> ERROR executor.Executor: Exception in task ID 0
>>> java.lang.ClassNotFoundException: $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
>>
>>
>> It is my guess that for some reason Spark is not able to find the
>> anonymous function (_ % 2 == 1). Please note I'm running locally so I did
>> not provide any jars. For some reason when using first() instead of count()
>> it works. Needless to say it also works in Spark Shell but as I stated,
>> working with it is not an option.
>>
>> This issue brings much sadness to my heart, and I could not find a
>> solution on the mailing list archives or elsewhere. I am hoping someone
>> here might offer some help.
>>
>> Thanks,
>> Jon
>>
>
>
Re: Using Spark in IntelliJ Scala Console
Posted by Michael Armbrust <mi...@databricks.com>.
The spark-shell is a special version of the Scala REPL that serves the
classes created for each line over HTTP. Do you know if the InteliJ Spark
console is just the normal Scala repl in a GUI wrapper, or if it is
something else entirely? If its the former, perhaps it might be possible
to tell InteliJ to bring up the spark version instead.
On Sat, Apr 26, 2014 at 10:47 AM, Jonathan Chayat <
jonathan.c@supersonicads.com> wrote:
> Hi all,
>
> TLDR: running spark locally through IntelliJ IDEA Scala Console results
> in java.lang.ClassNotFoundException
>
> Long version:
>
> I'm an algorithms developer in SupersonicAds - an ad network. We are
> building a major new big data project and we are now in the process of
> selecting our tech stack & tools.
>
> I'm new to Spark, but I'm very excited about it. It is my opinion that
> Spark can be a great tool for us, and that we might be able to build most
> of our toolchain on top of it.
>
> We currently develop in Scala and we are using IntelliJ IDEA as our IDE
> (we love it). One of the features I love about IDEA is the Scala Console
> which lets me work interactively with all of my project's code available
> and all of the IDE's features & convenience. That is as opposed to the
> Scala Shell & Spark Shell which I dislike because it is based on JLine and
> doesn't behave like a good shell would (I cant even Ctrl-c to abort a line
> without crashing the whole thing). Of course, as an algo guy, having a good
> REPL is crucial to me.
>
> To get started, I added the following line to build.sbt:
>
>> "org.apache.spark" %% "spark-core" % "0.9.1"
>
>
> Then, added the following main class:
>
> import org.apache.spark.SparkContext
>> import org.apache.spark.SparkContext._
>>
>
>
> object Main extends App {
>> val sc = new SparkContext("local", "myApp")
>> val r = sc.parallelize(1 to 1000)
>> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 ==
>> 1).first())
>> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 ==
>> 1).count())
>> }
>
>
> Make, Run, Works perfectly.
>
> Next, I try running the same in the scala console.
> Bad news - the last line throws an exception:
>
>> ERROR executor.Executor: Exception in task ID 0
>> java.lang.ClassNotFoundException: $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
>
>
> It is my guess that for some reason Spark is not able to find the
> anonymous function (_ % 2 == 1). Please note I'm running locally so I did
> not provide any jars. For some reason when using first() instead of count()
> it works. Needless to say it also works in Spark Shell but as I stated,
> working with it is not an option.
>
> This issue brings much sadness to my heart, and I could not find a
> solution on the mailing list archives or elsewhere. I am hoping someone
> here might offer some help.
>
> Thanks,
> Jon
>