You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Laurent Hatier <la...@gmail.com> on 2011/05/27 13:52:04 UTC
Hadoop problem
Hi,
I'm a newbie with Hadoop/MapReduce. I've a problem with hadoop. I set some
variables in the run function but when Map running, he can't get the value
of theses variables...
If anyone knows the solution :)
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
Re: Hadoop problem
Posted by Laurent Hatier <la...@gmail.com>.
I have written a mistake : of course SpecificRecord is in the cassandra API
but my question is : why i have this problem ? or Is the an another way to
write back the result ?
Thanks
2011/5/30 Laurent Hatier <la...@gmail.com>
> Hi everybody,
>
> I have a little problem with cassandra-all jar file : when i want to write
> back the result of the MapReduce in DB, he says me that the SpecificRecord
> class (Hector API) is not found... I have already check this dependency and
> it's ok. Do I have to use the Cassandra API or it's a technical problem ?
>
> Thanks
>
>
> 2011/5/27 Laurent Hatier <la...@gmail.com>
>
>> Of couurse !!! It's logical.
>> Thank you John.
>>
>> 2011/5/27 John Armstrong <jo...@ccri.com>
>>
>>> On Fri, 27 May 2011 13:52:04 +0200, Laurent Hatier
>>> <la...@gmail.com> wrote:
>>> > I'm a newbie with Hadoop/MapReduce. I've a problem with hadoop. I set
>>> some
>>> > variables in the run function but when Map running, he can't get the
>>> value
>>> > of theses variables...
>>> > If anyone knows the solution :)
>>>
>>> By the "run function" do you mean the main method that launches the
>>> map/reduce job? It's no surprise that the mappers (and reducers) won't
>>> know those variables, because they run as completely separate tasks.
>>>
>>> If you're computing something in the setup method for use in the mappers
>>> or reducers you'll have to pass that information along somehow. If it's
>>> a
>>> String (or something that can easily be made into a String, like an int)
>>> you can set it as a property in the job's Configuration. For more
>>> complicated data you'll have to serialize it to a file, place the file
>>> into
>>> the distributed cache, and then deserialize the data within the mapper or
>>> reducer's setup method.
>>>
>>> Of course, if the computation is less complicated/time consuming than the
>>> deserialization process, you may as well just recompute the data in each
>>> mapper or reducer.
>>>
>>
>>
>>
>> --
>> Laurent HATIER
>> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>>
>
>
>
> --
> Laurent HATIER
> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
Re: Hadoop problem
Posted by Laurent Hatier <la...@gmail.com>.
Hi everybody,
I have a little problem with cassandra-all jar file : when i want to write
back the result of the MapReduce in DB, he says me that the SpecificRecord
class (Hector API) is not found... I have already check this dependency and
it's ok. Do I have to use the Cassandra API or it's a technical problem ?
Thanks
2011/5/27 Laurent Hatier <la...@gmail.com>
> Of couurse !!! It's logical.
> Thank you John.
>
> 2011/5/27 John Armstrong <jo...@ccri.com>
>
>> On Fri, 27 May 2011 13:52:04 +0200, Laurent Hatier
>> <la...@gmail.com> wrote:
>> > I'm a newbie with Hadoop/MapReduce. I've a problem with hadoop. I set
>> some
>> > variables in the run function but when Map running, he can't get the
>> value
>> > of theses variables...
>> > If anyone knows the solution :)
>>
>> By the "run function" do you mean the main method that launches the
>> map/reduce job? It's no surprise that the mappers (and reducers) won't
>> know those variables, because they run as completely separate tasks.
>>
>> If you're computing something in the setup method for use in the mappers
>> or reducers you'll have to pass that information along somehow. If it's a
>> String (or something that can easily be made into a String, like an int)
>> you can set it as a property in the job's Configuration. For more
>> complicated data you'll have to serialize it to a file, place the file
>> into
>> the distributed cache, and then deserialize the data within the mapper or
>> reducer's setup method.
>>
>> Of course, if the computation is less complicated/time consuming than the
>> deserialization process, you may as well just recompute the data in each
>> mapper or reducer.
>>
>
>
>
> --
> Laurent HATIER
> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
Re: Hadoop problem
Posted by Laurent Hatier <la...@gmail.com>.
Of couurse !!! It's logical.
Thank you John.
2011/5/27 John Armstrong <jo...@ccri.com>
> On Fri, 27 May 2011 13:52:04 +0200, Laurent Hatier
> <la...@gmail.com> wrote:
> > I'm a newbie with Hadoop/MapReduce. I've a problem with hadoop. I set
> some
> > variables in the run function but when Map running, he can't get the
> value
> > of theses variables...
> > If anyone knows the solution :)
>
> By the "run function" do you mean the main method that launches the
> map/reduce job? It's no surprise that the mappers (and reducers) won't
> know those variables, because they run as completely separate tasks.
>
> If you're computing something in the setup method for use in the mappers
> or reducers you'll have to pass that information along somehow. If it's a
> String (or something that can easily be made into a String, like an int)
> you can set it as a property in the job's Configuration. For more
> complicated data you'll have to serialize it to a file, place the file into
> the distributed cache, and then deserialize the data within the mapper or
> reducer's setup method.
>
> Of course, if the computation is less complicated/time consuming than the
> deserialization process, you may as well just recompute the data in each
> mapper or reducer.
>
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
Re: Hadoop problem
Posted by John Armstrong <jo...@ccri.com>.
On Fri, 27 May 2011 13:52:04 +0200, Laurent Hatier
<la...@gmail.com> wrote:
> I'm a newbie with Hadoop/MapReduce. I've a problem with hadoop. I set
some
> variables in the run function but when Map running, he can't get the
value
> of theses variables...
> If anyone knows the solution :)
By the "run function" do you mean the main method that launches the
map/reduce job? It's no surprise that the mappers (and reducers) won't
know those variables, because they run as completely separate tasks.
If you're computing something in the setup method for use in the mappers
or reducers you'll have to pass that information along somehow. If it's a
String (or something that can easily be made into a String, like an int)
you can set it as a property in the job's Configuration. For more
complicated data you'll have to serialize it to a file, place the file into
the distributed cache, and then deserialize the data within the mapper or
reducer's setup method.
Of course, if the computation is less complicated/time consuming than the
deserialization process, you may as well just recompute the data in each
mapper or reducer.