You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sandeep Singh <sa...@techaddict.me> on 2014/09/11 15:04:04 UTC
Spark on Raspberry Pi?
Has anyone tried using Raspberry Pi for Spark? How efficient is it to use
around 10 Pi's for local testing env ?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Raspberry-Pi-tp13965.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on Raspberry Pi?
Posted by Chen He <ai...@gmail.com>.
Here is the POC of cubieboards hadoop cluster performance and power
consumption.
I suggest cubietrunk would be a choice since cubieboard2 only has 100Mbps
ethernet.
http://www.slideshare.net/airbots/hadoop-mapreduce-performance-study-on-arm-cluster
On Thu, Sep 11, 2014 at 12:50 PM, Chanwit Kaewkasi <ch...@gmail.com>
wrote:
> We've found that Raspberry Pi is not enough for Hadoop/Spark mainly
> because the memory consumption. What we've built is a cluster form
> with 22 Cubieboards, each contains 1 GB RAM.
>
> Best regards,
>
> -chanwit
>
> --
> Chanwit Kaewkasi
> linkedin.com/in/chanwit
>
>
> On Thu, Sep 11, 2014 at 8:04 PM, Sandeep Singh <sa...@techaddict.me>
> wrote:
> > Has anyone tried using Raspberry Pi for Spark? How efficient is it to use
> > around 10 Pi's for local testing env ?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Raspberry-Pi-tp13965.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark on Raspberry Pi?
Posted by Chanwit Kaewkasi <ch...@gmail.com>.
We've found that Raspberry Pi is not enough for Hadoop/Spark mainly
because the memory consumption. What we've built is a cluster form
with 22 Cubieboards, each contains 1 GB RAM.
Best regards,
-chanwit
--
Chanwit Kaewkasi
linkedin.com/in/chanwit
On Thu, Sep 11, 2014 at 8:04 PM, Sandeep Singh <sa...@techaddict.me> wrote:
> Has anyone tried using Raspberry Pi for Spark? How efficient is it to use
> around 10 Pi's for local testing env ?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Raspberry-Pi-tp13965.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on Raspberry Pi?
Posted by Aniket Bhatnagar <an...@gmail.com>.
Just curiois... What's the use case you are looking to implement?
On Sep 11, 2014 10:50 PM, "Daniil Osipov" <da...@shazam.com> wrote:
> Limited memory could also cause you some problems and limit usability. If
> you're looking for a local testing environment, vagrant boxes may serve you
> much better.
>
> On Thu, Sep 11, 2014 at 6:18 AM, Chen He <ai...@gmail.com> wrote:
>
>>
>>
>>
>> Pi's bus speed, memory size and access speed, and processing ability are
>> limited. The only benefit could be the power consumption.
>>
>> On Thu, Sep 11, 2014 at 8:04 AM, Sandeep Singh <sa...@techaddict.me>
>> wrote:
>>
>>> Has anyone tried using Raspberry Pi for Spark? How efficient is it to use
>>> around 10 Pi's for local testing env ?
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Raspberry-Pi-tp13965.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>>
>
Re: Spark on Raspberry Pi?
Posted by Daniil Osipov <da...@shazam.com>.
Limited memory could also cause you some problems and limit usability. If
you're looking for a local testing environment, vagrant boxes may serve you
much better.
On Thu, Sep 11, 2014 at 6:18 AM, Chen He <ai...@gmail.com> wrote:
>
>
>
> Pi's bus speed, memory size and access speed, and processing ability are
> limited. The only benefit could be the power consumption.
>
> On Thu, Sep 11, 2014 at 8:04 AM, Sandeep Singh <sa...@techaddict.me>
> wrote:
>
>> Has anyone tried using Raspberry Pi for Spark? How efficient is it to use
>> around 10 Pi's for local testing env ?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Raspberry-Pi-tp13965.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
unable to create new native thread
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi
I am trying the Spark sample program “SparkPi”, I got an error "unable to create new native thread", how to resolve this?
14/09/11 21:36:16 INFO scheduler.DAGScheduler: Completed ResultTask(0, 644)
14/09/11 21:36:16 INFO scheduler.TaskSetManager: Finished TID 643 in 43 ms on node1 (progress: 636/100000)
14/09/11 21:36:16 INFO scheduler.DAGScheduler: Completed ResultTask(0, 643)
14/09/11 21:36:16 ERROR actor.ActorSystemImpl: Uncaught fatal error from thread [spark-akka.actor.default-dispatcher-16] shutting down ActorSystem [spark]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672)
at scala.concurrent.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1966)
at scala.concurrent.forkjoin.ForkJoinPool.externalPush(ForkJoinPool.java:1829)
at scala.concurrent.forkjoin.ForkJoinPool.execute(ForkJoinPool.java:2955)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool.execute(AbstractDispatcher.scala:374)
at akka.dispatch.ExecutorServiceDelegate$class.execute(ThreadPoolBuilder.scala:212)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:43)
at akka.dispatch.Dispatcher.registerForExecution(Dispatcher.scala:118)
at akka.dispatch.Dispatcher.dispatch(Dispatcher.scala:59)
at akka.actor.dungeon.Dispatch$class.sendMessage(Dispatch.scala:120)
at akka.actor.ActorCell.sendMessage(ActorCell.scala:338)
at akka.actor.Cell$class.sendMessage(ActorCell.scala:259)
at akka.actor.ActorCell.sendMessage(ActorCell.scala:338)
at akka.actor.LocalActorRef.$bang(ActorRef.scala:389)
at akka.actor.ActorRef.tell(ActorRef.scala:125)
at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:489)
at akka.actor.ActorCell.invoke(ActorCell.scala:455)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Regards
Arthur
Fwd: Spark on Raspberry Pi?
Posted by Chen He <ai...@gmail.com>.
Pi's bus speed, memory size and access speed, and processing ability are
limited. The only benefit could be the power consumption.
On Thu, Sep 11, 2014 at 8:04 AM, Sandeep Singh <sa...@techaddict.me>
wrote:
> Has anyone tried using Raspberry Pi for Spark? How efficient is it to use
> around 10 Pi's for local testing env ?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Raspberry-Pi-tp13965.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>