You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "bit1129@163.com" <bi...@163.com> on 2015/02/17 06:47:53 UTC

Question about spark streaming+Flume

Hi,
I am trying Spark Streaming + Flume example:

1. Code
object SparkFlumeNGExample { 
   def main(args : Array[String]) { 
   val conf = new SparkConf().setAppName("SparkFlumeNGExample") 
   val ssc = new StreamingContext(conf, Seconds(10)) 

   val lines = FlumeUtils.createStream(ssc,"localhost",9999) 
    // Print out the count of events received from this server in each batch 
   lines.count().map(cnt => "Received " + cnt + " flume events. at " + System.currentTimeMillis() ).print() 
   ssc.start() 
   ssc.awaitTermination(); 
} 
}
2. I submit the application with following sh:
./spark-submit --deploy-mode client --name SparkFlumeEventCount --master spark://hadoop.master:7077 --executor-memory 512M --total-executor-cores 2 --class spark.examples.streaming.SparkFlumeNGWordCount spark-streaming-flume.jar


When I write data to flume, I only notice the following console information that input is added.
storage.BlockManagerInfo: Added input-0-1424151807400 in memory on localhost:39338 (size: 1095.0 B, free: 267.2 MB) 
15/02/17 00:43:30 INFO scheduler.JobScheduler: Added jobs for time 1424151810000 ms 
15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time 1424151820000 ms
....
15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time 1424151870000 ms

But I didn't the output from the code: "Received X flumes events"

I am no idea where the problem is, any idea? Thanks






Re: Re: Question about spark streaming+Flume

Posted by "bit1129@163.com" <bi...@163.com>.
Thanks Arush..
With your code, compiling error occurs:

Error:(19, 11) value forechRDD is not a member of org.apache.spark.streaming.dstream.ReceiverInputDStream[org.apache.spark.streaming.flume.SparkFlumeEvent] 
lines.forechRDD(_.foreach(println)) 
^




From: Arush Kharbanda
Date: 2015-02-17 14:31
To: bit1129@163.com
CC: user
Subject: Re: Question about spark streaming+Flume
Hi

Can you try this

val lines = FlumeUtils.createStream(ssc,"localhost",9999) 
    // Print out the count of events received from this server in each batch 
   lines.count().map(cnt => "Received " + cnt + " flume events. at " + System.currentTimeMillis() )

lines.forechRDD(_.foreach(println))

Thanks
Arush

On Tue, Feb 17, 2015 at 11:17 AM, bit1129@163.com <bi...@163.com> wrote:
Hi,
I am trying Spark Streaming + Flume example:

1. Code
object SparkFlumeNGExample { 
   def main(args : Array[String]) { 
   val conf = new SparkConf().setAppName("SparkFlumeNGExample") 
   val ssc = new StreamingContext(conf, Seconds(10)) 

   val lines = FlumeUtils.createStream(ssc,"localhost",9999) 
    // Print out the count of events received from this server in each batch 
   lines.count().map(cnt => "Received " + cnt + " flume events. at " + System.currentTimeMillis() ).print() 
   ssc.start() 
   ssc.awaitTermination(); 
} 
}
2. I submit the application with following sh:
./spark-submit --deploy-mode client --name SparkFlumeEventCount --master spark://hadoop.master:7077 --executor-memory 512M --total-executor-cores 2 --class spark.examples.streaming.SparkFlumeNGWordCount spark-streaming-flume.jar


When I write data to flume, I only notice the following console information that input is added.
storage.BlockManagerInfo: Added input-0-1424151807400 in memory on localhost:39338 (size: 1095.0 B, free: 267.2 MB) 
15/02/17 00:43:30 INFO scheduler.JobScheduler: Added jobs for time 1424151810000 ms 
15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time 1424151820000 ms
....
15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time 1424151870000 ms

But I didn't the output from the code: "Received X flumes events"

I am no idea where the problem is, any idea? Thanks








-- 
Arush Kharbanda || Technical Teamlead
arush@sigmoidanalytics.com || www.sigmoidanalytics.com

Re: Re: Question about spark streaming+Flume

Posted by "bit1129@163.com" <bi...@163.com>.
Hi Arush, 
With your code, I still didn't see the output  "Received X flumes events"..



bit1129@163.com
 
From: bit1129@163.com
Date: 2015-02-17 14:08
To: Arush Kharbanda
CC: user
Subject: Re: Re: Question about spark streaming+Flume
Ok, you are missing a letter in foreachRDD.. let me proceed..



bit1129@163.com
 
From: Arush Kharbanda
Date: 2015-02-17 14:31
To: bit1129@163.com
CC: user
Subject: Re: Question about spark streaming+Flume
Hi

Can you try this

val lines = FlumeUtils.createStream(ssc,"localhost",9999) 
    // Print out the count of events received from this server in each batch 
   lines.count().map(cnt => "Received " + cnt + " flume events. at " + System.currentTimeMillis() )

lines.forechRDD(_.foreach(println))

Thanks
Arush

On Tue, Feb 17, 2015 at 11:17 AM, bit1129@163.com <bi...@163.com> wrote:
Hi,
I am trying Spark Streaming + Flume example:

1. Code
object SparkFlumeNGExample { 
   def main(args : Array[String]) { 
   val conf = new SparkConf().setAppName("SparkFlumeNGExample") 
   val ssc = new StreamingContext(conf, Seconds(10)) 

   val lines = FlumeUtils.createStream(ssc,"localhost",9999) 
    // Print out the count of events received from this server in each batch 
   lines.count().map(cnt => "Received " + cnt + " flume events. at " + System.currentTimeMillis() ).print() 
   ssc.start() 
   ssc.awaitTermination(); 
} 
}
2. I submit the application with following sh:
./spark-submit --deploy-mode client --name SparkFlumeEventCount --master spark://hadoop.master:7077 --executor-memory 512M --total-executor-cores 2 --class spark.examples.streaming.SparkFlumeNGWordCount spark-streaming-flume.jar


When I write data to flume, I only notice the following console information that input is added.
storage.BlockManagerInfo: Added input-0-1424151807400 in memory on localhost:39338 (size: 1095.0 B, free: 267.2 MB) 
15/02/17 00:43:30 INFO scheduler.JobScheduler: Added jobs for time 1424151810000 ms 
15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time 1424151820000 ms
....
15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time 1424151870000 ms

But I didn't the output from the code: "Received X flumes events"

I am no idea where the problem is, any idea? Thanks








-- 
Arush Kharbanda || Technical Teamlead
arush@sigmoidanalytics.com || www.sigmoidanalytics.com

Re: Re: Question about spark streaming+Flume

Posted by "bit1129@163.com" <bi...@163.com>.
Ok, you are missing a letter in foreachRDD.. let me proceed..



bit1129@163.com
 
From: Arush Kharbanda
Date: 2015-02-17 14:31
To: bit1129@163.com
CC: user
Subject: Re: Question about spark streaming+Flume
Hi

Can you try this

val lines = FlumeUtils.createStream(ssc,"localhost",9999) 
    // Print out the count of events received from this server in each batch 
   lines.count().map(cnt => "Received " + cnt + " flume events. at " + System.currentTimeMillis() )

lines.forechRDD(_.foreach(println))

Thanks
Arush

On Tue, Feb 17, 2015 at 11:17 AM, bit1129@163.com <bi...@163.com> wrote:
Hi,
I am trying Spark Streaming + Flume example:

1. Code
object SparkFlumeNGExample { 
   def main(args : Array[String]) { 
   val conf = new SparkConf().setAppName("SparkFlumeNGExample") 
   val ssc = new StreamingContext(conf, Seconds(10)) 

   val lines = FlumeUtils.createStream(ssc,"localhost",9999) 
    // Print out the count of events received from this server in each batch 
   lines.count().map(cnt => "Received " + cnt + " flume events. at " + System.currentTimeMillis() ).print() 
   ssc.start() 
   ssc.awaitTermination(); 
} 
}
2. I submit the application with following sh:
./spark-submit --deploy-mode client --name SparkFlumeEventCount --master spark://hadoop.master:7077 --executor-memory 512M --total-executor-cores 2 --class spark.examples.streaming.SparkFlumeNGWordCount spark-streaming-flume.jar


When I write data to flume, I only notice the following console information that input is added.
storage.BlockManagerInfo: Added input-0-1424151807400 in memory on localhost:39338 (size: 1095.0 B, free: 267.2 MB) 
15/02/17 00:43:30 INFO scheduler.JobScheduler: Added jobs for time 1424151810000 ms 
15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time 1424151820000 ms
....
15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time 1424151870000 ms

But I didn't the output from the code: "Received X flumes events"

I am no idea where the problem is, any idea? Thanks








-- 
Arush Kharbanda || Technical Teamlead
arush@sigmoidanalytics.com || www.sigmoidanalytics.com

Re: Question about spark streaming+Flume

Posted by Arush Kharbanda <ar...@sigmoidanalytics.com>.
Hi

Can you try this

val lines = FlumeUtils.createStream(ssc,"localhost",9999)
    // Print out the count of events received from this server in each
batch
   lines.count().map(cnt => "Received " + cnt + " flume events. at " +
System.currentTimeMillis() )

lines.forechRDD(_.foreach(println))

Thanks
Arush

On Tue, Feb 17, 2015 at 11:17 AM, bit1129@163.com <bi...@163.com> wrote:

> Hi,
> I am trying Spark Streaming + Flume example:
>
> 1. Code
> object SparkFlumeNGExample {
>    def main(args : Array[String]) {
>    val conf = new SparkConf().setAppName("SparkFlumeNGExample")
>    val ssc = new StreamingContext(conf, Seconds(10))
>
>    val lines = FlumeUtils.createStream(ssc,"localhost",9999)
>     // Print out the count of events received from this server in each
> batch
>    lines.count().map(cnt => "Received " + cnt + " flume events. at " +
> System.currentTimeMillis() ).print()
>    ssc.start()
>    ssc.awaitTermination();
> }
> }
> 2. I submit the application with following sh:
> ./spark-submit --deploy-mode client --name SparkFlumeEventCount --master
> spark://hadoop.master:7077 --executor-memory 512M --total-executor-cores 2
> --class spark.examples.streaming.SparkFlumeNGWordCount
> spark-streaming-flume.jar
>
>
> When I write data to flume, I only notice the following console
> information that input is added.
> storage.BlockManagerInfo: Added input-0-1424151807400 in memory on
> localhost:39338 (size: 1095.0 B, free: 267.2 MB)
> 15/02/17 00:43:30 INFO scheduler.JobScheduler: Added jobs for time
> 1424151810000 ms
> 15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time
> 1424151820000 ms
> ....
> 15/02/17 00:43:40 INFO scheduler.JobScheduler: Added jobs for time
> 1424151870000 ms
>
> But I didn't the output from the code: "Received X flumes events"
>
> I am no idea where the problem is, any idea? Thanks
>
>
> ------------------------------
>
>


-- 

[image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>

*Arush Kharbanda* || Technical Teamlead

arush@sigmoidanalytics.com || www.sigmoidanalytics.com