You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Suraj Satishkumar Sheth <su...@adobe.com> on 2014/03/04 10:20:03 UTC
RE: Actors and sparkcontext actions
Hi Ognen,
See if this helps. I was working on this :
class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], hdfsPath : String) extends Actor {
def act(){
if(flag1) this.process()
else this.count
}
private def process(){
println(sc.textFile(hdfsPath).count)
//do the processing
}
private def count(){
println(rdd.count)
//do the counting
}
}
Thanks and Regards,
Suraj Sheth
-----Original Message-----
From: Ognen Duzlevski [mailto:ognen@nengoiksvelzud.com]
Sent: 27 February 2014 01:09
To: user@spark.incubator.apache.org
Subject: Actors and sparkcontext actions
Can someone point me to a simple, short code example of creating a basic Actor that gets a context and runs an operation such as .textFile.count?
I am trying to figure out how to create just a basic actor that gets a message like this:
case class Msg(filename:String, ctx: SparkContext)
and then something like this:
class HelloActor extends Actor {
import context.dispatcher
def receive = {
case Msg(fn,ctx) => {
// get the count here!
// cts.textFile(fn).count
}
case _ => println("huh?")
}
}
Where I would want to do something like:
val conf = new
SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
val sc = new SparkContext(conf)
val system = ActorSystem("mySystem")
val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1")
helloActor1 ! new Msg("test.json",sc)
Thanks,
Ognen
Re: Actors and sparkcontext actions
Posted by Ognen Duzlevski <og...@plainvanillagames.com>.
Deb,
On 3/4/14, 9:02 AM, Debasish Das wrote:
> Hi Ognen,
>
> Any particular reason of choosing scalatra over options like play or
> spray ?
>
> Is scalatra much better in serving apis or is it due to similarity
> with ruby's sinatra ?
>
> Did you try the other options and then pick scalatra ?
Not really. I just happen to like Scalatra, it is easy with read and
easy to write. If it did not work out I would have gone probably with
something like Unfiltered.
Ognen
Re: Actors and sparkcontext actions
Posted by Debasish Das <de...@gmail.com>.
Hi Ognen,
Any particular reason of choosing scalatra over options like play or spray
?
Is scalatra much better in serving apis or is it due to similarity with
ruby's sinatra ?
Did you try the other options and then pick scalatra ?
Thanks.
Deb
On Tue, Mar 4, 2014 at 4:50 AM, Ognen Duzlevski <ognen@plainvanillagames.com
> wrote:
> Suraj, I posted to this list a link to my blog where I detail how to do a
> simple actor/sparkcontext thing with the added obstacle of it being within
> a Scalatra servlet.
>
> Thanks for the code!
> Ognen
>
>
> On 3/4/14, 3:20 AM, Suraj Satishkumar Sheth wrote:
>
>> Hi Ognen,
>> See if this helps. I was working on this :
>>
>> class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T],
>> hdfsPath : String) extends Actor {
>>
>> def act(){
>> if(flag1) this.process()
>> else this.count
>> }
>> private def process(){
>> println(sc.textFile(hdfsPath).count)
>> //do the processing
>> }
>> private def count(){
>> println(rdd.count)
>> //do the counting
>> }
>>
>> }
>>
>> Thanks and Regards,
>> Suraj Sheth
>>
>>
>> -----Original Message-----
>> From: Ognen Duzlevski [mailto:ognen@nengoiksvelzud.com]
>> Sent: 27 February 2014 01:09
>> To: user@spark.incubator.apache.org
>> Subject: Actors and sparkcontext actions
>>
>> Can someone point me to a simple, short code example of creating a basic
>> Actor that gets a context and runs an operation such as .textFile.count?
>> I am trying to figure out how to create just a basic actor that gets a
>> message like this:
>>
>> case class Msg(filename:String, ctx: SparkContext)
>>
>> and then something like this:
>>
>> class HelloActor extends Actor {
>> import context.dispatcher
>>
>> def receive = {
>> case Msg(fn,ctx) => {
>> // get the count here!
>> // cts.textFile(fn).count
>> }
>> case _ => println("huh?")
>> }
>> }
>>
>> Where I would want to do something like:
>>
>> val conf = new
>> SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").
>> setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
>> val sc = new SparkContext(conf)
>> val system = ActorSystem("mySystem")
>>
>> val helloActor1 = system.actorOf( Props[ HelloActor], name =
>> "helloactor1")
>> helloActor1 ! new Msg("test.json",sc)
>>
>> Thanks,
>> Ognen
>>
>
> --
> Some people, when confronted with a problem, think "I know, I'll use
> regular expressions." Now they have two problems.
> -- Jamie Zawinski
>
>
Re: Actors and sparkcontext actions
Posted by Ognen Duzlevski <og...@plainvanillagames.com>.
Suraj, I posted to this list a link to my blog where I detail how to do
a simple actor/sparkcontext thing with the added obstacle of it being
within a Scalatra servlet.
Thanks for the code!
Ognen
On 3/4/14, 3:20 AM, Suraj Satishkumar Sheth wrote:
> Hi Ognen,
> See if this helps. I was working on this :
>
> class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], hdfsPath : String) extends Actor {
>
> def act(){
> if(flag1) this.process()
> else this.count
> }
>
> private def process(){
> println(sc.textFile(hdfsPath).count)
> //do the processing
> }
>
> private def count(){
> println(rdd.count)
> //do the counting
> }
>
> }
>
> Thanks and Regards,
> Suraj Sheth
>
>
> -----Original Message-----
> From: Ognen Duzlevski [mailto:ognen@nengoiksvelzud.com]
> Sent: 27 February 2014 01:09
> To: user@spark.incubator.apache.org
> Subject: Actors and sparkcontext actions
>
> Can someone point me to a simple, short code example of creating a basic Actor that gets a context and runs an operation such as .textFile.count?
> I am trying to figure out how to create just a basic actor that gets a message like this:
>
> case class Msg(filename:String, ctx: SparkContext)
>
> and then something like this:
>
> class HelloActor extends Actor {
> import context.dispatcher
>
> def receive = {
> case Msg(fn,ctx) => {
> // get the count here!
> // cts.textFile(fn).count
> }
> case _ => println("huh?")
> }
> }
>
> Where I would want to do something like:
>
> val conf = new
> SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
> val sc = new SparkContext(conf)
> val system = ActorSystem("mySystem")
>
> val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1")
> helloActor1 ! new Msg("test.json",sc)
>
> Thanks,
> Ognen
--
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
-- Jamie Zawinski