You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Jaonary Rabarisoa <ja...@gmail.com> on 2014/04/11 14:38:12 UTC

Hybrid GPU CPU computation

Hi all,

I'm just wondering if hybrid GPU/CPU computation is something that is
feasible with spark ? And what should be the best way to do it.


Cheers,

Jaonary

Re: Hybrid GPU CPU computation

Posted by Jaonary Rabarisoa <ja...@gmail.com>.

In fact the idea is to run some part of the code on GPU as Patrick
described and extend the RDD structure so that it can also be distributed
on GPU's. The following article
http://www.wired.com/2013/06/andrew_ng/ describes a hybrid GPU/GPU
implementation (with MPI) that outperforms a
16, 000 cores cluster.

On Fri, Apr 11, 2014 at 3:53 PM, Patrick Grinaway <pg...@gmail.com>wrote:

> I've actually done it using PySpark and python libraries which call cuda
> code, though I've never done it from scala directly. The only major
> challenge I've hit is assigning tasks to gpus on multiple gpu machines.
>
> Sent from my iPhone
>
> > On Apr 11, 2014, at 8:38 AM, Jaonary Rabarisoa <ja...@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I'm just wondering if hybrid GPU/CPU computation is something that is
> feasible with spark ? And what should be the best way to do it.
> >
> >
> > Cheers,
> >
> > Jaonary
>

Re: Hybrid GPU CPU computation

Posted by Patrick Grinaway <pg...@gmail.com>.

I've actually done it using PySpark and python libraries which call cuda code, though I've never done it from scala directly. The only major challenge I've hit is assigning tasks to gpus on multiple gpu machines. 

Sent from my iPhone

> On Apr 11, 2014, at 8:38 AM, Jaonary Rabarisoa <ja...@gmail.com> wrote:
> 
> Hi all,
> 
> I'm just wondering if hybrid GPU/CPU computation is something that is feasible with spark ? And what should be the best way to do it.
> 
> 
> Cheers,
> 
> Jaonary

Re: Hybrid GPU CPU computation

Posted by Pascal Voitot Dev <pa...@gmail.com>.

On Fri, Apr 11, 2014 at 3:34 PM, Dean Wampler <de...@gmail.com> wrote:

> I've thought about this idea, although I haven't tried it, but I think the
> right approach is to pick your granularity boundary and use Spark + JVM for
> large-scale parts of the algorithm, then use the gpgus API for number
> crunching large chunks at a time. No need to run the JVM and Spark on the
> GPU, which would make no sense anyway.
>
>
I find that would be crazy to be able to run the JVM on a GPU even if it's
a bit non-sense XD
Anyway, you're right, the approach by delegating just some parts of the
code to the GPU is interesting but it also means you have to pre-install
this code on all cluster nodes...


> Here's another approach:
> http://www.cakesolutions.net/teamblogs/2013/02/13/akka-and-cuda/
>
> dean
>
>
> On Fri, Apr 11, 2014 at 7:49 AM, Saurabh Jha <sa...@gmail.com>wrote:
>
>> There is a scala implementation for gpgus (nvidia cuda to be precise).
>> but you also need to port mesos for gpu's. I am not sure about mesos. Also,
>> the current scala gpu version is not stable to be used commercially.
>>
>> Hope this helps.
>>
>> Thanks
>> saurabh.
>>
>>
>>
>> *Saurabh Jha*
>> Intl. Exchange Student
>> School of Computing Engineering
>> Nanyang Technological University,
>> Singapore
>> Web: http://profile.saurabhjha.in
>> Mob: +65 94663172
>>
>>
>> On Fri, Apr 11, 2014 at 8:40 PM, Pascal Voitot Dev <
>> pascal.voitot.dev@gmail.com> wrote:
>>
>>> This is a bit crazy :)
>>> I suppose you would have to run Java code on the GPU!
>>> I heard there are some funny projects to do that...
>>>
>>> Pascal
>>>
>>> On Fri, Apr 11, 2014 at 2:38 PM, Jaonary Rabarisoa <ja...@gmail.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm just wondering if hybrid GPU/CPU computation is something that is
>>>> feasible with spark ? And what should be the best way to do it.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Jaonary
>>>>
>>>
>>>
>>
>
>
> --
> Dean Wampler, Ph.D.
> Typesafe
> @deanwampler
> http://typesafe.com
> http://polyglotprogramming.com
>

Re: Hybrid GPU CPU computation

Posted by Dean Wampler <de...@gmail.com>.

I've thought about this idea, although I haven't tried it, but I think the
right approach is to pick your granularity boundary and use Spark + JVM for
large-scale parts of the algorithm, then use the gpgus API for number
crunching large chunks at a time. No need to run the JVM and Spark on the
GPU, which would make no sense anyway.

Here's another approach:
http://www.cakesolutions.net/teamblogs/2013/02/13/akka-and-cuda/

dean


On Fri, Apr 11, 2014 at 7:49 AM, Saurabh Jha <sa...@gmail.com>wrote:

> There is a scala implementation for gpgus (nvidia cuda to be precise). but
> you also need to port mesos for gpu's. I am not sure about mesos. Also, the
> current scala gpu version is not stable to be used commercially.
>
> Hope this helps.
>
> Thanks
> saurabh.
>
>
>
> *Saurabh Jha*
> Intl. Exchange Student
> School of Computing Engineering
> Nanyang Technological University,
> Singapore
> Web: http://profile.saurabhjha.in
> Mob: +65 94663172
>
>
> On Fri, Apr 11, 2014 at 8:40 PM, Pascal Voitot Dev <
> pascal.voitot.dev@gmail.com> wrote:
>
>> This is a bit crazy :)
>> I suppose you would have to run Java code on the GPU!
>> I heard there are some funny projects to do that...
>>
>> Pascal
>>
>> On Fri, Apr 11, 2014 at 2:38 PM, Jaonary Rabarisoa <ja...@gmail.com>wrote:
>>
>>> Hi all,
>>>
>>> I'm just wondering if hybrid GPU/CPU computation is something that is
>>> feasible with spark ? And what should be the best way to do it.
>>>
>>>
>>> Cheers,
>>>
>>> Jaonary
>>>
>>
>>
>


-- 
Dean Wampler, Ph.D.
Typesafe
@deanwampler
http://typesafe.com
http://polyglotprogramming.com

Re: Hybrid GPU CPU computation

Posted by Saurabh Jha <sa...@gmail.com>.

There is a scala implementation for gpgus (nvidia cuda to be precise). but
you also need to port mesos for gpu's. I am not sure about mesos. Also, the
current scala gpu version is not stable to be used commercially.

Hope this helps.

Thanks
saurabh.

*Saurabh Jha*
Intl. Exchange Student
School of Computing Engineering
Nanyang Technological University,
Singapore
Web: http://profile.saurabhjha.in
Mob: +65 94663172

On Fri, Apr 11, 2014 at 8:40 PM, Pascal Voitot Dev <
pascal.voitot.dev@gmail.com> wrote:

> This is a bit crazy :)
> I suppose you would have to run Java code on the GPU!
> I heard there are some funny projects to do that...
>
> Pascal
>
> On Fri, Apr 11, 2014 at 2:38 PM, Jaonary Rabarisoa <ja...@gmail.com>wrote:
>
>> Hi all,
>>
>> I'm just wondering if hybrid GPU/CPU computation is something that is
>> feasible with spark ? And what should be the best way to do it.
>>
>>
>> Cheers,
>>
>> Jaonary
>>
>
>

Re: Hybrid GPU CPU computation

Posted by Pascal Voitot Dev <pa...@gmail.com>.

This is a bit crazy :)
I suppose you would have to run Java code on the GPU!
I heard there are some funny projects to do that...

Pascal

On Fri, Apr 11, 2014 at 2:38 PM, Jaonary Rabarisoa <ja...@gmail.com>wrote:

> Hi all,
>
> I'm just wondering if hybrid GPU/CPU computation is something that is
> feasible with spark ? And what should be the best way to do it.
>
>
> Cheers,
>
> Jaonary
>