You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Adarsh Sharma <ad...@orkash.com> on 2011/03/04 06:18:45 UTC
Cuda Program in Hadoop Cluster
Dear all,
I followed a fantastic tutorial and able to run the Wordcont C++ Program
in Hadoop Cluster.
http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_2.2_--_Running_C%2B%2B_Programs_on_Hadoop
But know I want to run a Cuda Program in the Hadoop Cluster but results
in errors.
Is anyone has done it before and guide me how to do this.
I attached the both files. Please find the attachment.
Thanks & best Regards,
Adarsh Sharma
Re: Cuda Program in Hadoop Cluster
Posted by Russell Jurney <ru...@gmail.com>.
A Hadoop/CUDA example would be great, and probably of general interest
to the list.
Russ
Sent from my iPhone
On Mar 13, 2011, at 4:14 AM, "pdu@cse.unl.edu" <pd...@cse.unl.edu> wrote:
> Hello Adarsh,
>
> At the time I was doing this integration, I used Pipes because it's C++
> based, which is easier for porting CUDA C code I assume.
>
> It actually is quite straightforward I think. What I have done can be
> summarized into:
>
> 1) a simple abstract class that implements the common protocol for
> Mapper/Reducer such that each CUDA mapper/reducer does not have to deal
> with the Pipes communication. They just launch CUDA kernel and collect
> results.
>
> 2) a set of simple shell scripts for uploading the mapper/reducer/CUDA
> kernel to the Hadoop DFS and kickstarts the runs.
>
> 3) modify the CUDA program into a mapper inheriting from the wrapper class
> so that it takes input from Pipes and dump results also to Pipes.
>
> If you want, I can try to find the code for you but I think you get the
> idea. I didn't do a whole lot because it's mainly a class project. We just
> want to demonstrate its feasibility and power saving.
>
> Thank you for your interest in our work! Let me know if you need further
> assistance.
>
>
> Peng
>> Thanks Chen for your guidance,
>>
>> To translate CUDA in Hadoop environment, following are the options :
>>
>> -- Hadoop Streaming: Standard I/O ( Recommended by you )
>> -- Hadoop Pipes: C++ library, Socket connection, ( I'm looking for )
>> -- JNI, JNI-based CUDA wrapper (JCUDA) ( Done and able to run a Jcuda
>> Program in Hadoop Cluster )
>>
>> We use Hadoop Pipes for our proposed technique
>> . MapReduce applications/CUDA kernel ? written in C++
>>
>> So i.e why I am trying a C Cuda code to run through Hadoop Pipes, is it
>> difficult or not possible.
>> I am looking for a demo Cuda program that is up and running in Hadoop
>> Cluster that clarifies my basic concepts so that I program accordingly
>> in future.
>>
>>
>> Looking forward for some more guidance.
>>
>> Thanks once again for your wishes .
>>
>>
>> With best Regards,
>>
>> Adarsh Sharma
>>
>>
>>
>> He Chen wrote:
>>> Hi, Adarsh Sharma
>>>
>>> For C code
>>>
>>> My friend employ hadoop streaming to run CUDA C code. You can send email
>>> to
>>> him. pdu@cse.unl.edu.
>>>
>>> Best wishes!
>>>
>>> Chen
>>>
>>>
>>> On Thu, Mar 3, 2011 at 11:18 PM, Adarsh Sharma
>>> <ad...@orkash.com>wrote:
>>>
>>>
>>>> Dear all,
>>>>
>>>> I followed a fantastic tutorial and able to run the Wordcont C++
>>>> Program in
>>>> Hadoop Cluster.
>>>>
>>>>
>>>> http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_2.2_--_Running_C%2B%2B_Programs_on_Hadoop
>>>>
>>>> But know I want to run a Cuda Program in the Hadoop Cluster but results
>>>> in
>>>> errors.
>>>> Is anyone has done it before and guide me how to do this.
>>>>
>>>> I attached the both files. Please find the attachment.
>>>>
>>>>
>>>> Thanks & best Regards,
>>>>
>>>> Adarsh Sharma
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: Cuda Program in Hadoop Cluster
Posted by pd...@cse.unl.edu.
Hello Adarsh,
At the time I was doing this integration, I used Pipes because it's C++
based, which is easier for porting CUDA C code I assume.
It actually is quite straightforward I think. What I have done can be
summarized into:
1) a simple abstract class that implements the common protocol for
Mapper/Reducer such that each CUDA mapper/reducer does not have to deal
with the Pipes communication. They just launch CUDA kernel and collect
results.
2) a set of simple shell scripts for uploading the mapper/reducer/CUDA
kernel to the Hadoop DFS and kickstarts the runs.
3) modify the CUDA program into a mapper inheriting from the wrapper class
so that it takes input from Pipes and dump results also to Pipes.
If you want, I can try to find the code for you but I think you get the
idea. I didn't do a whole lot because it's mainly a class project. We just
want to demonstrate its feasibility and power saving.
Thank you for your interest in our work! Let me know if you need further
assistance.
Peng
> Thanks Chen for your guidance,
>
> To translate CUDA in Hadoop environment, following are the options :
>
> -- Hadoop Streaming: Standard I/O ( Recommended by you )
> -- Hadoop Pipes: C++ library, Socket connection, ( I'm looking for )
> -- JNI, JNI-based CUDA wrapper (JCUDA) ( Done and able to run a Jcuda
> Program in Hadoop Cluster )
>
> We use Hadoop Pipes for our proposed technique
> . MapReduce applications/CUDA kernel ? written in C++
>
> So i.e why I am trying a C Cuda code to run through Hadoop Pipes, is it
> difficult or not possible.
> I am looking for a demo Cuda program that is up and running in Hadoop
> Cluster that clarifies my basic concepts so that I program accordingly
> in future.
>
>
> Looking forward for some more guidance.
>
> Thanks once again for your wishes .
>
>
> With best Regards,
>
> Adarsh Sharma
>
>
>
> He Chen wrote:
>> Hi, Adarsh Sharma
>>
>> For C code
>>
>> My friend employ hadoop streaming to run CUDA C code. You can send email
>> to
>> him. pdu@cse.unl.edu.
>>
>> Best wishes!
>>
>> Chen
>>
>>
>> On Thu, Mar 3, 2011 at 11:18 PM, Adarsh Sharma
>> <ad...@orkash.com>wrote:
>>
>>
>>> Dear all,
>>>
>>> I followed a fantastic tutorial and able to run the Wordcont C++
>>> Program in
>>> Hadoop Cluster.
>>>
>>>
>>> http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_2.2_--_Running_C%2B%2B_Programs_on_Hadoop
>>>
>>> But know I want to run a Cuda Program in the Hadoop Cluster but results
>>> in
>>> errors.
>>> Is anyone has done it before and guide me how to do this.
>>>
>>> I attached the both files. Please find the attachment.
>>>
>>>
>>> Thanks & best Regards,
>>>
>>> Adarsh Sharma
>>>
>>>
>>
>>
>
>
Re: Cuda Program in Hadoop Cluster
Posted by Adarsh Sharma <ad...@orkash.com>.
So, it means it is impossible to run GPU code ( myfile.cu ) through
Hadoop Pipes .
It's the requirement to run a C++ code that includes some Cuda code (
Cuda libraries & _global_ function ) in a Hadoop Cluster.
Thanks & best Regards,
Adarsh Sharma
Lance Norskog wrote:
> One of the Python CUDA packages translates Python byte-code into
> CUDA-style C and runs the toolchain. If this actually works, you can
> just do Python apps under Hadoop Streaming.
>
> On Wed, Mar 9, 2011 at 8:39 PM, Adarsh Sharma <ad...@orkash.com> wrote:
>
>> Thanks Chen for your guidance,
>>
>> To translate CUDA in Hadoop environment, following are the options :
>>
>> -- Hadoop Streaming: Standard I/O ( Recommended by you )
>> -- Hadoop Pipes: C++ library, Socket connection, ( I'm looking for )
>> -- JNI, JNI-based CUDA wrapper (JCUDA) ( Done and able to run a Jcuda
>> Program in Hadoop Cluster )
>>
>> We use Hadoop Pipes for our proposed technique
>> . MapReduce applications/CUDA kernel ? written in C++
>>
>> So i.e why I am trying a C Cuda code to run through Hadoop Pipes, is it
>> difficult or not possible.
>> I am looking for a demo Cuda program that is up and running in Hadoop
>> Cluster that clarifies my basic concepts so that I program accordingly in
>> future.
>>
>>
>> Looking forward for some more guidance.
>>
>> Thanks once again for your wishes .
>>
>>
>> With best Regards,
>>
>> Adarsh Sharma
>>
>>
>>
>> He Chen wrote:
>>
>>> Hi, Adarsh Sharma
>>>
>>> For C code
>>>
>>> My friend employ hadoop streaming to run CUDA C code. You can send email
>>> to
>>> him. pdu@cse.unl.edu.
>>>
>>> Best wishes!
>>>
>>> Chen
>>>
>>>
>>> On Thu, Mar 3, 2011 at 11:18 PM, Adarsh Sharma
>>> <ad...@orkash.com>wrote:
>>>
>>>
>>>
>>>> Dear all,
>>>>
>>>> I followed a fantastic tutorial and able to run the Wordcont C++ Program
>>>> in
>>>> Hadoop Cluster.
>>>>
>>>>
>>>>
>>>> http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_2.2_--_Running_C%2B%2B_Programs_on_Hadoop
>>>>
>>>> But know I want to run a Cuda Program in the Hadoop Cluster but results
>>>> in
>>>> errors.
>>>> Is anyone has done it before and guide me how to do this.
>>>>
>>>> I attached the both files. Please find the attachment.
>>>>
>>>>
>>>> Thanks & best Regards,
>>>>
>>>> Adarsh Sharma
>>>>
>>>>
>>>>
>>>
>>
>
>
>
>
Re: Cuda Program in Hadoop Cluster
Posted by Lance Norskog <go...@gmail.com>.
One of the Python CUDA packages translates Python byte-code into
CUDA-style C and runs the toolchain. If this actually works, you can
just do Python apps under Hadoop Streaming.
On Wed, Mar 9, 2011 at 8:39 PM, Adarsh Sharma <ad...@orkash.com> wrote:
> Thanks Chen for your guidance,
>
> To translate CUDA in Hadoop environment, following are the options :
>
> -- Hadoop Streaming: Standard I/O ( Recommended by you )
> -- Hadoop Pipes: C++ library, Socket connection, ( I'm looking for )
> -- JNI, JNI-based CUDA wrapper (JCUDA) ( Done and able to run a Jcuda
> Program in Hadoop Cluster )
>
> We use Hadoop Pipes for our proposed technique
> . MapReduce applications/CUDA kernel ? written in C++
>
> So i.e why I am trying a C Cuda code to run through Hadoop Pipes, is it
> difficult or not possible.
> I am looking for a demo Cuda program that is up and running in Hadoop
> Cluster that clarifies my basic concepts so that I program accordingly in
> future.
>
>
> Looking forward for some more guidance.
>
> Thanks once again for your wishes .
>
>
> With best Regards,
>
> Adarsh Sharma
>
>
>
> He Chen wrote:
>>
>> Hi, Adarsh Sharma
>>
>> For C code
>>
>> My friend employ hadoop streaming to run CUDA C code. You can send email
>> to
>> him. pdu@cse.unl.edu.
>>
>> Best wishes!
>>
>> Chen
>>
>>
>> On Thu, Mar 3, 2011 at 11:18 PM, Adarsh Sharma
>> <ad...@orkash.com>wrote:
>>
>>
>>>
>>> Dear all,
>>>
>>> I followed a fantastic tutorial and able to run the Wordcont C++ Program
>>> in
>>> Hadoop Cluster.
>>>
>>>
>>>
>>> http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_2.2_--_Running_C%2B%2B_Programs_on_Hadoop
>>>
>>> But know I want to run a Cuda Program in the Hadoop Cluster but results
>>> in
>>> errors.
>>> Is anyone has done it before and guide me how to do this.
>>>
>>> I attached the both files. Please find the attachment.
>>>
>>>
>>> Thanks & best Regards,
>>>
>>> Adarsh Sharma
>>>
>>>
>>
>>
>
>
--
Lance Norskog
goksron@gmail.com
Re: Cuda Program in Hadoop Cluster
Posted by Adarsh Sharma <ad...@orkash.com>.
Thanks Chen for your guidance,
To translate CUDA in Hadoop environment, following are the options :
-- Hadoop Streaming: Standard I/O ( Recommended by you )
-- Hadoop Pipes: C++ library, Socket connection, ( I'm looking for )
-- JNI, JNI-based CUDA wrapper (JCUDA) ( Done and able to run a Jcuda
Program in Hadoop Cluster )
We use Hadoop Pipes for our proposed technique
. MapReduce applications/CUDA kernel ? written in C++
So i.e why I am trying a C Cuda code to run through Hadoop Pipes, is it
difficult or not possible.
I am looking for a demo Cuda program that is up and running in Hadoop
Cluster that clarifies my basic concepts so that I program accordingly
in future.
Looking forward for some more guidance.
Thanks once again for your wishes .
With best Regards,
Adarsh Sharma
He Chen wrote:
> Hi, Adarsh Sharma
>
> For C code
>
> My friend employ hadoop streaming to run CUDA C code. You can send email to
> him. pdu@cse.unl.edu.
>
> Best wishes!
>
> Chen
>
>
> On Thu, Mar 3, 2011 at 11:18 PM, Adarsh Sharma <ad...@orkash.com>wrote:
>
>
>> Dear all,
>>
>> I followed a fantastic tutorial and able to run the Wordcont C++ Program in
>> Hadoop Cluster.
>>
>>
>> http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_2.2_--_Running_C%2B%2B_Programs_on_Hadoop
>>
>> But know I want to run a Cuda Program in the Hadoop Cluster but results in
>> errors.
>> Is anyone has done it before and guide me how to do this.
>>
>> I attached the both files. Please find the attachment.
>>
>>
>> Thanks & best Regards,
>>
>> Adarsh Sharma
>>
>>
>
>
Re: Cuda Program in Hadoop Cluster
Posted by He Chen <ai...@gmail.com>.
Hi, Adarsh Sharma
For C code
My friend employ hadoop streaming to run CUDA C code. You can send email to
him. pdu@cse.unl.edu.
Best wishes!
Chen
On Thu, Mar 3, 2011 at 11:18 PM, Adarsh Sharma <ad...@orkash.com>wrote:
> Dear all,
>
> I followed a fantastic tutorial and able to run the Wordcont C++ Program in
> Hadoop Cluster.
>
>
> http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_2.2_--_Running_C%2B%2B_Programs_on_Hadoop
>
> But know I want to run a Cuda Program in the Hadoop Cluster but results in
> errors.
> Is anyone has done it before and guide me how to do this.
>
> I attached the both files. Please find the attachment.
>
>
> Thanks & best Regards,
>
> Adarsh Sharma
>