You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com> on 2013/08/14 06:35:46 UTC
Calling a MATLAB library in map reduce program
Hi,
I have to run some analytics on the files present in HDFS using a MATLAB code. I am thinking of compiling the MATLAB code into a C++ library and calling it in map reduce code. How can I implement this? I read Hadoop streaming or Hadoop pipes can be used for this. But I have not tried it on my own. Please share your valuable suggestions.
Regards,
Anand.C
RE: Calling a MATLAB library in map reduce program
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
Thanks for all the suggestion. I will explore more and raise specific questions if needed.
Regards,
Anand.C
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, August 15, 2013 1:30 AM
To: user@hadoop.apache.org
Subject: Re: Calling a MATLAB library in map reduce program
To add to that, if you want to take advantage of MapReduce, e.g. you need to do a distributed grouping or sort, pipes or streaming would be the way to go. If you're mainly interested in running your code in parallel on a cluster, distributed shell, a YARN application outside of MapReduce, could be a simpler way to accomplish this.
-Sandy
On Wed, Aug 14, 2013 at 11:17 AM, Omkar Joshi <oj...@hortonworks.com>> wrote:
Can you take a look at distributed shell example? It will be helpful.
Thanks,
Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>
On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <An...@honeywell.com>> wrote:
Hi,
I have to run some analytics on the files present in HDFS using a MATLAB code. I am thinking of compiling the MATLAB code into a C++ library and calling it in map reduce code. How can I implement this? I read Hadoop streaming or Hadoop pipes can be used for this. But I have not tried it on my own. Please share your valuable suggestions.
Regards,
Anand.C
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Calling a MATLAB library in map reduce program
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
Thanks for all the suggestion. I will explore more and raise specific questions if needed.
Regards,
Anand.C
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, August 15, 2013 1:30 AM
To: user@hadoop.apache.org
Subject: Re: Calling a MATLAB library in map reduce program
To add to that, if you want to take advantage of MapReduce, e.g. you need to do a distributed grouping or sort, pipes or streaming would be the way to go. If you're mainly interested in running your code in parallel on a cluster, distributed shell, a YARN application outside of MapReduce, could be a simpler way to accomplish this.
-Sandy
On Wed, Aug 14, 2013 at 11:17 AM, Omkar Joshi <oj...@hortonworks.com>> wrote:
Can you take a look at distributed shell example? It will be helpful.
Thanks,
Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>
On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <An...@honeywell.com>> wrote:
Hi,
I have to run some analytics on the files present in HDFS using a MATLAB code. I am thinking of compiling the MATLAB code into a C++ library and calling it in map reduce code. How can I implement this? I read Hadoop streaming or Hadoop pipes can be used for this. But I have not tried it on my own. Please share your valuable suggestions.
Regards,
Anand.C
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Calling a MATLAB library in map reduce program
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
Thanks for all the suggestion. I will explore more and raise specific questions if needed.
Regards,
Anand.C
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, August 15, 2013 1:30 AM
To: user@hadoop.apache.org
Subject: Re: Calling a MATLAB library in map reduce program
To add to that, if you want to take advantage of MapReduce, e.g. you need to do a distributed grouping or sort, pipes or streaming would be the way to go. If you're mainly interested in running your code in parallel on a cluster, distributed shell, a YARN application outside of MapReduce, could be a simpler way to accomplish this.
-Sandy
On Wed, Aug 14, 2013 at 11:17 AM, Omkar Joshi <oj...@hortonworks.com>> wrote:
Can you take a look at distributed shell example? It will be helpful.
Thanks,
Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>
On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <An...@honeywell.com>> wrote:
Hi,
I have to run some analytics on the files present in HDFS using a MATLAB code. I am thinking of compiling the MATLAB code into a C++ library and calling it in map reduce code. How can I implement this? I read Hadoop streaming or Hadoop pipes can be used for this. But I have not tried it on my own. Please share your valuable suggestions.
Regards,
Anand.C
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Calling a MATLAB library in map reduce program
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
Thanks for all the suggestion. I will explore more and raise specific questions if needed.
Regards,
Anand.C
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, August 15, 2013 1:30 AM
To: user@hadoop.apache.org
Subject: Re: Calling a MATLAB library in map reduce program
To add to that, if you want to take advantage of MapReduce, e.g. you need to do a distributed grouping or sort, pipes or streaming would be the way to go. If you're mainly interested in running your code in parallel on a cluster, distributed shell, a YARN application outside of MapReduce, could be a simpler way to accomplish this.
-Sandy
On Wed, Aug 14, 2013 at 11:17 AM, Omkar Joshi <oj...@hortonworks.com>> wrote:
Can you take a look at distributed shell example? It will be helpful.
Thanks,
Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>
On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <An...@honeywell.com>> wrote:
Hi,
I have to run some analytics on the files present in HDFS using a MATLAB code. I am thinking of compiling the MATLAB code into a C++ library and calling it in map reduce code. How can I implement this? I read Hadoop streaming or Hadoop pipes can be used for this. But I have not tried it on my own. Please share your valuable suggestions.
Regards,
Anand.C
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Calling a MATLAB library in map reduce program
Posted by Sandy Ryza <sa...@cloudera.com>.
To add to that, if you want to take advantage of MapReduce, e.g. you need
to do a distributed grouping or sort, pipes or streaming would be the way
to go. If you're mainly interested in running your code in parallel on a
cluster, distributed shell, a YARN application outside of MapReduce, could
be a simpler way to accomplish this.
-Sandy
On Wed, Aug 14, 2013 at 11:17 AM, Omkar Joshi <oj...@hortonworks.com>wrote:
> Can you take a look at distributed shell example? It will be helpful.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <
> Ananda.Murugan@honeywell.com> wrote:
>
>> Hi,****
>>
>> ** **
>>
>> I have to run some analytics on the files present in HDFS using a MATLAB
>> code. I am thinking of compiling the MATLAB code into a C++ library and
>> calling it in map reduce code. How can I implement this? I read Hadoop
>> streaming or Hadoop pipes can be used for this. But I have not tried it on
>> my own. Please share your valuable suggestions. ****
>>
>> ** **
>>
>> Regards,****
>>
>> Anand.C****
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Calling a MATLAB library in map reduce program
Posted by Sandy Ryza <sa...@cloudera.com>.
To add to that, if you want to take advantage of MapReduce, e.g. you need
to do a distributed grouping or sort, pipes or streaming would be the way
to go. If you're mainly interested in running your code in parallel on a
cluster, distributed shell, a YARN application outside of MapReduce, could
be a simpler way to accomplish this.
-Sandy
On Wed, Aug 14, 2013 at 11:17 AM, Omkar Joshi <oj...@hortonworks.com>wrote:
> Can you take a look at distributed shell example? It will be helpful.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <
> Ananda.Murugan@honeywell.com> wrote:
>
>> Hi,****
>>
>> ** **
>>
>> I have to run some analytics on the files present in HDFS using a MATLAB
>> code. I am thinking of compiling the MATLAB code into a C++ library and
>> calling it in map reduce code. How can I implement this? I read Hadoop
>> streaming or Hadoop pipes can be used for this. But I have not tried it on
>> my own. Please share your valuable suggestions. ****
>>
>> ** **
>>
>> Regards,****
>>
>> Anand.C****
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Calling a MATLAB library in map reduce program
Posted by Sandy Ryza <sa...@cloudera.com>.
To add to that, if you want to take advantage of MapReduce, e.g. you need
to do a distributed grouping or sort, pipes or streaming would be the way
to go. If you're mainly interested in running your code in parallel on a
cluster, distributed shell, a YARN application outside of MapReduce, could
be a simpler way to accomplish this.
-Sandy
On Wed, Aug 14, 2013 at 11:17 AM, Omkar Joshi <oj...@hortonworks.com>wrote:
> Can you take a look at distributed shell example? It will be helpful.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <
> Ananda.Murugan@honeywell.com> wrote:
>
>> Hi,****
>>
>> ** **
>>
>> I have to run some analytics on the files present in HDFS using a MATLAB
>> code. I am thinking of compiling the MATLAB code into a C++ library and
>> calling it in map reduce code. How can I implement this? I read Hadoop
>> streaming or Hadoop pipes can be used for this. But I have not tried it on
>> my own. Please share your valuable suggestions. ****
>>
>> ** **
>>
>> Regards,****
>>
>> Anand.C****
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Calling a MATLAB library in map reduce program
Posted by Sandy Ryza <sa...@cloudera.com>.
To add to that, if you want to take advantage of MapReduce, e.g. you need
to do a distributed grouping or sort, pipes or streaming would be the way
to go. If you're mainly interested in running your code in parallel on a
cluster, distributed shell, a YARN application outside of MapReduce, could
be a simpler way to accomplish this.
-Sandy
On Wed, Aug 14, 2013 at 11:17 AM, Omkar Joshi <oj...@hortonworks.com>wrote:
> Can you take a look at distributed shell example? It will be helpful.
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <
> Ananda.Murugan@honeywell.com> wrote:
>
>> Hi,****
>>
>> ** **
>>
>> I have to run some analytics on the files present in HDFS using a MATLAB
>> code. I am thinking of compiling the MATLAB code into a C++ library and
>> calling it in map reduce code. How can I implement this? I read Hadoop
>> streaming or Hadoop pipes can be used for this. But I have not tried it on
>> my own. Please share your valuable suggestions. ****
>>
>> ** **
>>
>> Regards,****
>>
>> Anand.C****
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Calling a MATLAB library in map reduce program
Posted by Omkar Joshi <oj...@hortonworks.com>.
Can you take a look at distributed shell example? It will be helpful.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi,****
>
> ** **
>
> I have to run some analytics on the files present in HDFS using a MATLAB
> code. I am thinking of compiling the MATLAB code into a C++ library and
> calling it in map reduce code. How can I implement this? I read Hadoop
> streaming or Hadoop pipes can be used for this. But I have not tried it on
> my own. Please share your valuable suggestions. ****
>
> ** **
>
> Regards,****
>
> Anand.C****
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Calling a MATLAB library in map reduce program
Posted by Omkar Joshi <oj...@hortonworks.com>.
Can you take a look at distributed shell example? It will be helpful.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi,****
>
> ** **
>
> I have to run some analytics on the files present in HDFS using a MATLAB
> code. I am thinking of compiling the MATLAB code into a C++ library and
> calling it in map reduce code. How can I implement this? I read Hadoop
> streaming or Hadoop pipes can be used for this. But I have not tried it on
> my own. Please share your valuable suggestions. ****
>
> ** **
>
> Regards,****
>
> Anand.C****
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Calling a MATLAB library in map reduce program
Posted by Omkar Joshi <oj...@hortonworks.com>.
Can you take a look at distributed shell example? It will be helpful.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi,****
>
> ** **
>
> I have to run some analytics on the files present in HDFS using a MATLAB
> code. I am thinking of compiling the MATLAB code into a C++ library and
> calling it in map reduce code. How can I implement this? I read Hadoop
> streaming or Hadoop pipes can be used for this. But I have not tried it on
> my own. Please share your valuable suggestions. ****
>
> ** **
>
> Regards,****
>
> Anand.C****
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Calling a MATLAB library in map reduce program
Posted by Omkar Joshi <oj...@hortonworks.com>.
Can you take a look at distributed shell example? It will be helpful.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Tue, Aug 13, 2013 at 9:35 PM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi,****
>
> ** **
>
> I have to run some analytics on the files present in HDFS using a MATLAB
> code. I am thinking of compiling the MATLAB code into a C++ library and
> calling it in map reduce code. How can I implement this? I read Hadoop
> streaming or Hadoop pipes can be used for this. But I have not tried it on
> my own. Please share your valuable suggestions. ****
>
> ** **
>
> Regards,****
>
> Anand.C****
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.