java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext

[Kal El <pi...@yahoo.com>]

I am trying to run this code on spark 0.9.0 and I am getting this error: "java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext"

My CLASSPATH variable contains the following path"....spark-0.9.0/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-hadoop1.0.4.jar"
what else do I need to add here ?

The code works fine in the spark-shell.

"import org.apache.spark.streaming.{Time, Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._

object HdfsWordCount {

  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println("Usage: HdfsWordCount <master> <directory>")
      System.exit(1)
    }

  //  StreamingExamples.setStreamingLogLevels()

    // Create the context
//    val ssc = new StreamingContext(args(0), "HdfsWordCount", Seconds(2), System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_HOME")))
    val ssc = new StreamingContext(args(0), "HdfsWordCount", Seconds(2))
    // Create the FileInputDStream on the directory and use the
    // stream to count words in new files created
    val lines = ssc.textFileStream(args(1))
    val words = lines.flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}"

Thanks

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext

[Amit Behera <am...@gmail.com>]

Hi Kal El,
to run you an application outside the spark-shell you need to write a *build
file* (example.sbt) parallel to your source code.
step1: *example.sbt*

name := "Simple Project"

version := "1.0"

scalaVersion := "2.9.3"

libraryDependencies += "org.apache.spark" %% "spark-­streaming" %
"0.9.0­-incubating"

resolvers ++= Seq( "Akka Repository" at
"http://repo.akka.io/releases/","Spray Repository" at
"http://repo.spray.cc/")

step2:

download sbt from
http://scalasbt.artifactoryonline.com/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.12.2/sbt.tgz

unzip it and set the path. (up to sbt/bin)

step3:

parallel to your .sbt file type command

$sbt package

$sbt run

[?]





On Tue, Feb 11, 2014 at 2:48 PM, Kal El <pi...@yahoo.com> wrote:

> I am trying to run this code on spark 0.9.0 and I am getting this error:
> "java.lang.ClassNotFoundException:
> org.apache.spark.streaming.StreamingContext"
>
> My CLASSPATH variable contains the following
> path"....spark-0.9.0/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-hadoop1.0.4.jar"
> what else do I need to add here ?
>
> The code works fine in the spark-shell.
>
> "import org.apache.spark.streaming.{Time, Seconds, StreamingContext}
> import org.apache.spark.streaming.StreamingContext._
>
> object HdfsWordCount {
>   def main(args: Array[String]) {
>     if (args.length < 2) {
>       System.err.println("Usage: HdfsWordCount <master> <directory>")
>       System.exit(1)
>     }
>
>   //  StreamingExamples.setStreamingLogLevels()
>
>     // Create the context
> //    val ssc = new StreamingContext(args(0), "HdfsWordCount", Seconds(2),
> System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_HOME")))
>     val ssc = new StreamingContext(args(0), "HdfsWordCount", Seconds(2))
>     // Create the FileInputDStream on the directory and use the
>     // stream to count words in new files created
>     val lines = ssc.textFileStream(args(1))
>     val words = lines.flatMap(_.split(" "))
>     val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
>     wordCounts.print()
>     ssc.start()
>     ssc.awaitTermination()
>   }
> }"
>
> Thanks
>
>
>