You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Lac Trung <tr...@gmail.com> on 2012/03/04 11:12:36 UTC
How does Hadoop compile the program written in language other than
Java ?
Hi everyone !
Hadoop is written in Java, so mapreduce programs are written in Java, too.
But Hadoop provides an API to MapReduce that allows you to write your map
and reduce functions in languages other than Java (ex. Python), called
Hadoop Streaming.
I read the guide of Hadoop Streaming in
here<http://www.hadoop.apache.org/common/docs/r0.15.2/streaming.html#More+usage+examples>
but
I haven't seen any paragraph that write about converting language to Java.
Can anybody tell me how Hadoop compile the program written in language
other than Java.
Thank you !
--
Lac Trung
Re: How does Hadoop compile the program written in language other
than Java ?
Posted by Gopal <ab...@gmail.com>.
On 3/4/2012 5:44 AM, Kai Voigt wrote:
> Hi,
>
> please read http://developer.yahoo.com/hadoop/tutorial/module4.html#streaming and http://developer.yahoo.com/hadoop/tutorial/module4.html#streaming for more explanation and examples.
>
> Kai
>
> Am 04.03.2012 um 16:10 schrieb Lac Trung:
>
>> Can you give one or some examples about this, Kai ?
>> I haven't understood how Hadoop run a mapreduce program in other language :D
>>
>> Vào 02:21 Ngày 04 tháng 3 năm 2012, Kai Voigt<k...@123.org> đã viết:
>>
>>> Hi,
>>>
>>> the streaming API doesn't compile the streaming scripts.
>>>
>>> The PHP/Perl/Python/Ruby scripts you create as mapper and reducer will be
>>> called as external programs.
>>>
>>> The input key/value pairs will be send to your scripts as stdin, and the
>>> output will be collected from their stdout.
>>>
>>> So, no compilation, the scripts will just be executed.
>>>
>>> Kai
>>>
>>> Am 04.03.2012 um 15:42 schrieb Lac Trung:
>>>
>>>> Hi everyone !
>>>>
>>>> Hadoop is written in Java, so mapreduce programs are written in Java,
>>> too.
>>>> But Hadoop provides an API to MapReduce that allows you to write your map
>>>> and reduce functions in languages other than Java (ex. Python), called
>>>> Hadoop Streaming.
>>>> I read the guide of Hadoop Streaming in
>>>> here<
>>> http://www.hadoop.apache.org/common/docs/r0.15.2/streaming.html#More+usage+examples
>>>> but
>>>> I haven't seen any paragraph that write about converting language to
>>> Java.
>>>> Can anybody tell me how Hadoop compile the program written in language
>>>> other than Java.
>>>>
>>>> Thank you !
>>>> --
>>>> Lac Trung
>>> --
>>> Kai Voigt
>>> k@123.org
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> Lạc Trung
>> 20083535
Its nothing out of ordinary.
The java routines written for your specific language ( Jruby, Jython )
etc. will interpret the code and run it.
No different from how Perl or Php work.
Thanks
Re: How does Hadoop compile the program written in language other
than Java ?
Posted by Lac Trung <tr...@gmail.com>.
Can anyone give me a diagram that describe the process from given input
data to output data ?
Vào 02:57 Ngày 04 tháng 3 năm 2012, Lac Trung <tr...@gmail.com> đã
viết:
> Oh, I read it twice but it is familiar with this<http://hadoop.apache.org/common/docs/r0.15.2/streaming.html>.
> Now I think I should read Hadoop Pipe first. You are so pro, Kai.
>
> Vào 02:44 Ngày 04 tháng 3 năm 2012, Kai Voigt <k...@123.org> đã viết:
>
> Hi,
>>
>> please read
>> http://developer.yahoo.com/hadoop/tutorial/module4.html#streaming and
>> http://developer.yahoo.com/hadoop/tutorial/module4.html#streaming for
>> more explanation and examples.
>>
>> Kai
>>
>> Am 04.03.2012 um 16:10 schrieb Lac Trung:
>>
>> > Can you give one or some examples about this, Kai ?
>> > I haven't understood how Hadoop run a mapreduce program in other
>> language :D
>> >
>> > Vào 02:21 Ngày 04 tháng 3 năm 2012, Kai Voigt <k...@123.org> đã viết:
>> >
>> >> Hi,
>> >>
>> >> the streaming API doesn't compile the streaming scripts.
>> >>
>> >> The PHP/Perl/Python/Ruby scripts you create as mapper and reducer will
>> be
>> >> called as external programs.
>> >>
>> >> The input key/value pairs will be send to your scripts as stdin, and
>> the
>> >> output will be collected from their stdout.
>> >>
>> >> So, no compilation, the scripts will just be executed.
>> >>
>> >> Kai
>> >>
>> >> Am 04.03.2012 um 15:42 schrieb Lac Trung:
>> >>
>> >>> Hi everyone !
>> >>>
>> >>> Hadoop is written in Java, so mapreduce programs are written in Java,
>> >> too.
>> >>> But Hadoop provides an API to MapReduce that allows you to write your
>> map
>> >>> and reduce functions in languages other than Java (ex. Python), called
>> >>> Hadoop Streaming.
>> >>> I read the guide of Hadoop Streaming in
>> >>> here<
>> >>
>> http://www.hadoop.apache.org/common/docs/r0.15.2/streaming.html#More+usage+examples
>> >>>
>> >>> but
>> >>> I haven't seen any paragraph that write about converting language to
>> >> Java.
>> >>> Can anybody tell me how Hadoop compile the program written in language
>> >>> other than Java.
>> >>>
>> >>> Thank you !
>> >>> --
>> >>> Lac Trung
>> >>
>> >> --
>> >> Kai Voigt
>> >> k@123.org
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Lạc Trung
>> > 20083535
>>
>> --
>> Kai Voigt
>> k@123.org
>>
>>
>>
>>
>>
>
>
> --
> Lạc Trung
> 20083535
>
>
--
Lạc Trung
20083535
Re: How does Hadoop compile the program written in language other
than Java ?
Posted by Lac Trung <tr...@gmail.com>.
Oh, I read it twice but it is familiar with
this<http://hadoop.apache.org/common/docs/r0.15.2/streaming.html>.
Now I think I should read Hadoop Pipe first. You are so pro, Kai.
Vào 02:44 Ngày 04 tháng 3 năm 2012, Kai Voigt <k...@123.org> đã viết:
> Hi,
>
> please read
> http://developer.yahoo.com/hadoop/tutorial/module4.html#streaming and
> http://developer.yahoo.com/hadoop/tutorial/module4.html#streaming for
> more explanation and examples.
>
> Kai
>
> Am 04.03.2012 um 16:10 schrieb Lac Trung:
>
> > Can you give one or some examples about this, Kai ?
> > I haven't understood how Hadoop run a mapreduce program in other
> language :D
> >
> > Vào 02:21 Ngày 04 tháng 3 năm 2012, Kai Voigt <k...@123.org> đã viết:
> >
> >> Hi,
> >>
> >> the streaming API doesn't compile the streaming scripts.
> >>
> >> The PHP/Perl/Python/Ruby scripts you create as mapper and reducer will
> be
> >> called as external programs.
> >>
> >> The input key/value pairs will be send to your scripts as stdin, and the
> >> output will be collected from their stdout.
> >>
> >> So, no compilation, the scripts will just be executed.
> >>
> >> Kai
> >>
> >> Am 04.03.2012 um 15:42 schrieb Lac Trung:
> >>
> >>> Hi everyone !
> >>>
> >>> Hadoop is written in Java, so mapreduce programs are written in Java,
> >> too.
> >>> But Hadoop provides an API to MapReduce that allows you to write your
> map
> >>> and reduce functions in languages other than Java (ex. Python), called
> >>> Hadoop Streaming.
> >>> I read the guide of Hadoop Streaming in
> >>> here<
> >>
> http://www.hadoop.apache.org/common/docs/r0.15.2/streaming.html#More+usage+examples
> >>>
> >>> but
> >>> I haven't seen any paragraph that write about converting language to
> >> Java.
> >>> Can anybody tell me how Hadoop compile the program written in language
> >>> other than Java.
> >>>
> >>> Thank you !
> >>> --
> >>> Lac Trung
> >>
> >> --
> >> Kai Voigt
> >> k@123.org
> >>
> >>
> >>
> >>
> >>
> >
> >
> > --
> > Lạc Trung
> > 20083535
>
> --
> Kai Voigt
> k@123.org
>
>
>
>
>
--
Lạc Trung
20083535
Re: How does Hadoop compile the program written in language other than Java ?
Posted by Kai Voigt <k...@123.org>.
Hi,
please read http://developer.yahoo.com/hadoop/tutorial/module4.html#streaming and http://developer.yahoo.com/hadoop/tutorial/module4.html#streaming for more explanation and examples.
Kai
Am 04.03.2012 um 16:10 schrieb Lac Trung:
> Can you give one or some examples about this, Kai ?
> I haven't understood how Hadoop run a mapreduce program in other language :D
>
> Vào 02:21 Ngày 04 tháng 3 năm 2012, Kai Voigt <k...@123.org> đã viết:
>
>> Hi,
>>
>> the streaming API doesn't compile the streaming scripts.
>>
>> The PHP/Perl/Python/Ruby scripts you create as mapper and reducer will be
>> called as external programs.
>>
>> The input key/value pairs will be send to your scripts as stdin, and the
>> output will be collected from their stdout.
>>
>> So, no compilation, the scripts will just be executed.
>>
>> Kai
>>
>> Am 04.03.2012 um 15:42 schrieb Lac Trung:
>>
>>> Hi everyone !
>>>
>>> Hadoop is written in Java, so mapreduce programs are written in Java,
>> too.
>>> But Hadoop provides an API to MapReduce that allows you to write your map
>>> and reduce functions in languages other than Java (ex. Python), called
>>> Hadoop Streaming.
>>> I read the guide of Hadoop Streaming in
>>> here<
>> http://www.hadoop.apache.org/common/docs/r0.15.2/streaming.html#More+usage+examples
>>>
>>> but
>>> I haven't seen any paragraph that write about converting language to
>> Java.
>>> Can anybody tell me how Hadoop compile the program written in language
>>> other than Java.
>>>
>>> Thank you !
>>> --
>>> Lac Trung
>>
>> --
>> Kai Voigt
>> k@123.org
>>
>>
>>
>>
>>
>
>
> --
> Lạc Trung
> 20083535
--
Kai Voigt
k@123.org
Re: How does Hadoop compile the program written in language other
than Java ?
Posted by Lac Trung <tr...@gmail.com>.
Can you give one or some examples about this, Kai ?
I haven't understood how Hadoop run a mapreduce program in other language :D
Vào 02:21 Ngày 04 tháng 3 năm 2012, Kai Voigt <k...@123.org> đã viết:
> Hi,
>
> the streaming API doesn't compile the streaming scripts.
>
> The PHP/Perl/Python/Ruby scripts you create as mapper and reducer will be
> called as external programs.
>
> The input key/value pairs will be send to your scripts as stdin, and the
> output will be collected from their stdout.
>
> So, no compilation, the scripts will just be executed.
>
> Kai
>
> Am 04.03.2012 um 15:42 schrieb Lac Trung:
>
> > Hi everyone !
> >
> > Hadoop is written in Java, so mapreduce programs are written in Java,
> too.
> > But Hadoop provides an API to MapReduce that allows you to write your map
> > and reduce functions in languages other than Java (ex. Python), called
> > Hadoop Streaming.
> > I read the guide of Hadoop Streaming in
> > here<
> http://www.hadoop.apache.org/common/docs/r0.15.2/streaming.html#More+usage+examples
> >
> > but
> > I haven't seen any paragraph that write about converting language to
> Java.
> > Can anybody tell me how Hadoop compile the program written in language
> > other than Java.
> >
> > Thank you !
> > --
> > Lac Trung
>
> --
> Kai Voigt
> k@123.org
>
>
>
>
>
--
Lạc Trung
20083535
Re: How does Hadoop compile the program written in language other than Java ?
Posted by Kai Voigt <k...@123.org>.
Hi,
the streaming API doesn't compile the streaming scripts.
The PHP/Perl/Python/Ruby scripts you create as mapper and reducer will be called as external programs.
The input key/value pairs will be send to your scripts as stdin, and the output will be collected from their stdout.
So, no compilation, the scripts will just be executed.
Kai
Am 04.03.2012 um 15:42 schrieb Lac Trung:
> Hi everyone !
>
> Hadoop is written in Java, so mapreduce programs are written in Java, too.
> But Hadoop provides an API to MapReduce that allows you to write your map
> and reduce functions in languages other than Java (ex. Python), called
> Hadoop Streaming.
> I read the guide of Hadoop Streaming in
> here<http://www.hadoop.apache.org/common/docs/r0.15.2/streaming.html#More+usage+examples>
> but
> I haven't seen any paragraph that write about converting language to Java.
> Can anybody tell me how Hadoop compile the program written in language
> other than Java.
>
> Thank you !
> --
> Lac Trung
--
Kai Voigt
k@123.org