You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Frederic Laruelle <fr...@gmail.com> on 2014/09/27 18:01:01 UTC
Map Reduce example for Python
I'm looking for a complete Map Reduce example for Python.
Particularly interested in how you connect to a Hadoop instance,
and how you write to and read from it.
Any pointers welcome.
Cheers,
Fred
Re: Map Reduce example for Python
Posted by Alex Kozlov <al...@gmail.com>.
You can also look at pyspark as an alternative to running MR with Python.
http://spark.apache.org/docs/1.1.0/programming-guide.html
On Sat, Sep 27, 2014 at 10:57 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Maybe this helps you out a bit.
> I am not sure how maintained the Avro wiki is, however the tutorial below
> would appear to cover most of the ground to get you up and running.
> hth
> Lewis
>
> https://cwiki.apache.org/confluence/display/AVRO/Using+AVRO+To+Run+Python+Map+Reduce+Jobs
>
> On Sat, Sep 27, 2014 at 9:01 AM, Frederic Laruelle <
> frederic.laruelle@gmail.com> wrote:
>
>> I'm looking for a complete Map Reduce example for Python.
>> Particularly interested in how you connect to a Hadoop instance,
>> and how you write to and read from it.
>> Any pointers welcome.
>>
>> Cheers,
>>
>> Fred
>>
>
>
>
> --
> *Lewis*
>
--
Alex Kozlov
(408) 507-4987
(408) 830-9982 fax
(650) 887-2135 efax
alexvk@gmail.com
Re: Map Reduce example for Python
Posted by Frederic Laruelle <fr...@gmail.com>.
Ha, actually,
You are giving me a good idea,
Ie look at the avro python implementation source code...
Tks Lewis :-)
> On Sep 27, 2014, at 11:47 AM, Lewis John Mcgibbney <le...@gmail.com> wrote:
>
> Hi Frederic,
> I honestly don't know however I doubt it as this does not feature in the trunk test base [0]
> I've honestly never used the Avro Python APi so I am not in a good position to provide you with any more info... sorry about that.
> Lewis
> [0] https://svn.apache.org/repos/asf/avro/trunk/lang/
>
>> On Sat, Sep 27, 2014 at 11:41 AM, Frederic Laruelle <fr...@gmail.com> wrote:
>> tks Lewis,
>> I did see that one already,
>> Is tethered process the only way to use Avro to serialize / deserialize objects to Hadoop?
>>
>>
>>> On Sat, Sep 27, 2014 at 10:57 AM, Lewis John Mcgibbney <le...@gmail.com> wrote:
>>> Maybe this helps you out a bit.
>>> I am not sure how maintained the Avro wiki is, however the tutorial below would appear to cover most of the ground to get you up and running.
>>> hth
>>> Lewis
>>> https://cwiki.apache.org/confluence/display/AVRO/Using+AVRO+To+Run+Python+Map+Reduce+Jobs
>>>
>>>> On Sat, Sep 27, 2014 at 9:01 AM, Frederic Laruelle <fr...@gmail.com> wrote:
>>>> I'm looking for a complete Map Reduce example for Python.
>>>> Particularly interested in how you connect to a Hadoop instance,
>>>> and how you write to and read from it.
>>>> Any pointers welcome.
>>>>
>>>> Cheers,
>>>>
>>>> Fred
>>>
>>>
>>>
>>> --
>>> Lewis
>
>
>
> --
> Lewis
Re: Map Reduce example for Python
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Frederic,
I honestly don't know however I doubt it as this does not feature in the
trunk test base [0]
I've honestly never used the Avro Python APi so I am not in a good position
to provide you with any more info... sorry about that.
Lewis
[0] https://svn.apache.org/repos/asf/avro/trunk/lang/
On Sat, Sep 27, 2014 at 11:41 AM, Frederic Laruelle <
frederic.laruelle@gmail.com> wrote:
> tks Lewis,
> I did see that one already,
> Is tethered process the only way to use Avro to serialize / deserialize
> objects to Hadoop?
>
>
> On Sat, Sep 27, 2014 at 10:57 AM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Maybe this helps you out a bit.
>> I am not sure how maintained the Avro wiki is, however the tutorial below
>> would appear to cover most of the ground to get you up and running.
>> hth
>> Lewis
>>
>> https://cwiki.apache.org/confluence/display/AVRO/Using+AVRO+To+Run+Python+Map+Reduce+Jobs
>>
>> On Sat, Sep 27, 2014 at 9:01 AM, Frederic Laruelle <
>> frederic.laruelle@gmail.com> wrote:
>>
>>> I'm looking for a complete Map Reduce example for Python.
>>> Particularly interested in how you connect to a Hadoop instance,
>>> and how you write to and read from it.
>>> Any pointers welcome.
>>>
>>> Cheers,
>>>
>>> Fred
>>>
>>
>>
>>
>> --
>> *Lewis*
>>
>
>
--
*Lewis*
Re: Map Reduce example for Python
Posted by Frederic Laruelle <fr...@gmail.com>.
tks Lewis,
I did see that one already,
Is tethered process the only way to use Avro to serialize / deserialize
objects to Hadoop?
On Sat, Sep 27, 2014 at 10:57 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Maybe this helps you out a bit.
> I am not sure how maintained the Avro wiki is, however the tutorial below
> would appear to cover most of the ground to get you up and running.
> hth
> Lewis
>
> https://cwiki.apache.org/confluence/display/AVRO/Using+AVRO+To+Run+Python+Map+Reduce+Jobs
>
> On Sat, Sep 27, 2014 at 9:01 AM, Frederic Laruelle <
> frederic.laruelle@gmail.com> wrote:
>
>> I'm looking for a complete Map Reduce example for Python.
>> Particularly interested in how you connect to a Hadoop instance,
>> and how you write to and read from it.
>> Any pointers welcome.
>>
>> Cheers,
>>
>> Fred
>>
>
>
>
> --
> *Lewis*
>
Re: Map Reduce example for Python
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Maybe this helps you out a bit.
I am not sure how maintained the Avro wiki is, however the tutorial below
would appear to cover most of the ground to get you up and running.
hth
Lewis
https://cwiki.apache.org/confluence/display/AVRO/Using+AVRO+To+Run+Python+Map+Reduce+Jobs
On Sat, Sep 27, 2014 at 9:01 AM, Frederic Laruelle <
frederic.laruelle@gmail.com> wrote:
> I'm looking for a complete Map Reduce example for Python.
> Particularly interested in how you connect to a Hadoop instance,
> and how you write to and read from it.
> Any pointers welcome.
>
> Cheers,
>
> Fred
>
--
*Lewis*