You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Sean Owen <sr...@gmail.com> on 2010/06/14 19:43:48 UTC
Is o.a.m.df.mapred.* removeable?
I'm about to make one of those big patches again. This time to push to
update to use the new Hadoop APIs everywhere (purely updating the API
calls).
Am I right that o.a.m.df.mapreduce.* duplicates and supersedes
o.a.m.df.mapred.*, the difference being that the former uses new APIs
already? I have this from an earlier message but want to confirm.
Sean
Re: Is o.a.m.df.mapred.* removeable?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
More to the point of your actual question, it looks to me like
o.a.m.df.mapreduce.* does superscede o.a.m.df.mapred.* as it employs the
new APIs as below.
On 6/14/10 11:22 AM, Jeff Eastman wrote:
> If you mean o.a.h.mapreduce.* then yes, mostly that. I found the
> drivers needed their JobConf stuff divided among Configuration and Job
> classes and the Mappers and Reducers needed new setup and cleanup
> methods with changes also due to the JobConf->Configuration/Job
> splits. Then the unit tests needed tweaking, particularly the Mapper
> and Reducer tests which needed MockMapperContexts and
> MockReducerContexts. Some of the driver stuff can be simplified by
> using the AbstractJob superclass methods and I will do that next in
> clustering.
>
> Have fun,
> Jeff
>
> On 6/14/10 10:43 AM, Sean Owen wrote:
>> I'm about to make one of those big patches again. This time to push to
>> update to use the new Hadoop APIs everywhere (purely updating the API
>> calls).
>>
>> Am I right that o.a.m.df.mapreduce.* duplicates and supersedes
>> o.a.m.df.mapred.*, the difference being that the former uses new APIs
>> already? I have this from an earlier message but want to confirm.
>>
>> Sean
>>
>
Re: Is o.a.m.df.mapred.* removeable?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
If you mean o.a.h.mapreduce.* then yes, mostly that. I found the drivers
needed their JobConf stuff divided among Configuration and Job classes
and the Mappers and Reducers needed new setup and cleanup methods with
changes also due to the JobConf->Configuration/Job splits. Then the unit
tests needed tweaking, particularly the Mapper and Reducer tests which
needed MockMapperContexts and MockReducerContexts. Some of the driver
stuff can be simplified by using the AbstractJob superclass methods and
I will do that next in clustering.
Have fun,
Jeff
On 6/14/10 10:43 AM, Sean Owen wrote:
> I'm about to make one of those big patches again. This time to push to
> update to use the new Hadoop APIs everywhere (purely updating the API
> calls).
>
> Am I right that o.a.m.df.mapreduce.* duplicates and supersedes
> o.a.m.df.mapred.*, the difference being that the former uses new APIs
> already? I have this from an earlier message but want to confirm.
>
> Sean
>
>
Re: Is o.a.m.df.mapred.* removeable?
Posted by Isabel Drost <is...@apache.org>.
On Mon Drew Farris <dr...@gmail.com> wrote:
> If I remember correctly there was some discussion at the beginning on
> April where it was proposed that the 'classic' apis in 0.20 would be
> undeprecated in the next release and that these would effectively
> become the 1.0 apis, with the new apis becoming the 2.0 apis, but
> I've lost track of the thread. I'm not aware as to whether a final
> decision was ever reached.
Is that the thread you mean?
http://www.mail-archive.com/mapreduce-dev@hadoop.apache.org/msg01833.html
https://issues.apache.org/jira/browse/MAPREDUCE-1735
To me this looks like the intention is to keep the old API around for
as long as the new one is still unstable. On the long term the new one
is going to be supported only.
Isabel
Re: Is o.a.m.df.mapred.* removeable?
Posted by Drew Farris <dr...@gmail.com>.
If I remember correctly there was some discussion at the beginning on April
where it was proposed that the 'classic' apis in 0.20 would be undeprecated
in the next release and that these would effectively become the 1.0 apis,
with the new apis becoming the 2.0 apis, but I've lost track of the thread.
I'm not aware as to whether a final decision was ever reached.
I have this vague recollection that the 'audience' annotations
(HADOOP-6668) were introduced with the intention of marking the 2.0 apis
'experimental', but I can't find a source for that.
On Mon, Jun 14, 2010 at 5:02 PM, Grant Ingersoll <gs...@apache.org>wrote:
> FWIW, someone at Buzzwords was telling me that Hadoop is undeprecating the
> old APIs in the next release. I find that even more confusing, but we might
> want to validate it and think about its implications.
>
>
> On Jun 14, 2010, at 1:43 PM, Sean Owen wrote:
>
> > I'm about to make one of those big patches again. This time to push to
> > update to use the new Hadoop APIs everywhere (purely updating the API
> > calls).
> >
> > Am I right that o.a.m.df.mapreduce.* duplicates and supersedes
> > o.a.m.df.mapred.*, the difference being that the former uses new APIs
> > already? I have this from an earlier message but want to confirm.
> >
> > Sean
>
>
>
Re: Is o.a.m.df.mapred.* removeable?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
I have the same problem in WikipediaDatasetCreatorOutputFormat in
examples..classifier.bayes. Might it be easier to port
MultipleOutputFormat?
The new APIs also don't let you set numMappers on the Job. I punted on
that (left a // TODO:) in FuzzyK but had to also in
WikipediaDatasetCreatorDriver.
On 6/15/10 10:36 AM, Sean Owen wrote:
> I've hit one big snag. MultipleOutputFormat doesn't exist anymore and
> Bayes uses it lots. I'm going to bail on this -- will have to be
> rewritten as many mapreduces.
>
> On Tue, Jun 15, 2010 at 4:17 PM, Jeff Eastman
> <jd...@windwardsolutions.com> wrote:
>
>> Ok, I'll start today on examples then move to utils. I will commit
>> everything before I head out Thursday.
>>
>> On 6/15/10 7:40 AM, Sean Owen wrote:
>>
>>> You want to do examples/utils? I can do core/ first, now. That's about
>>> an even split.
>>>
>>> On Mon, Jun 14, 2010 at 10:35 PM, Jeff Eastman
>>> <jd...@windwardsolutions.com> wrote:
>>>
>>>
>>>> +1 Having already done the port, I like the new APIs better than the old.
>>>> And I'm willing to help too if you want to carve out a few classes for
>>>> me. I
>>>> think I can do it in my sleep now. FWIW, I will be out again this
>>>> Thurs-Tues.
>>>>
>>>>
>>>
>>>
>>
>>
>
Re: Is o.a.m.df.mapred.* removeable?
Posted by Sean Owen <sr...@gmail.com>.
I've hit one big snag. MultipleOutputFormat doesn't exist anymore and
Bayes uses it lots. I'm going to bail on this -- will have to be
rewritten as many mapreduces.
On Tue, Jun 15, 2010 at 4:17 PM, Jeff Eastman
<jd...@windwardsolutions.com> wrote:
> Ok, I'll start today on examples then move to utils. I will commit
> everything before I head out Thursday.
>
> On 6/15/10 7:40 AM, Sean Owen wrote:
>>
>> You want to do examples/utils? I can do core/ first, now. That's about
>> an even split.
>>
>> On Mon, Jun 14, 2010 at 10:35 PM, Jeff Eastman
>> <jd...@windwardsolutions.com> wrote:
>>
>>>
>>> +1 Having already done the port, I like the new APIs better than the old.
>>> And I'm willing to help too if you want to carve out a few classes for
>>> me. I
>>> think I can do it in my sleep now. FWIW, I will be out again this
>>> Thurs-Tues.
>>>
>>
>>
>
>
Re: Is o.a.m.df.mapred.* removeable?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Ok, I'll start today on examples then move to utils. I will commit
everything before I head out Thursday.
On 6/15/10 7:40 AM, Sean Owen wrote:
> You want to do examples/utils? I can do core/ first, now. That's about
> an even split.
>
> On Mon, Jun 14, 2010 at 10:35 PM, Jeff Eastman
> <jd...@windwardsolutions.com> wrote:
>
>> +1 Having already done the port, I like the new APIs better than the old.
>> And I'm willing to help too if you want to carve out a few classes for me. I
>> think I can do it in my sleep now. FWIW, I will be out again this
>> Thurs-Tues.
>>
>
Re: Is o.a.m.df.mapred.* removeable?
Posted by Sean Owen <sr...@gmail.com>.
You want to do examples/utils? I can do core/ first, now. That's about
an even split.
On Mon, Jun 14, 2010 at 10:35 PM, Jeff Eastman
<jd...@windwardsolutions.com> wrote:
> +1 Having already done the port, I like the new APIs better than the old.
> And I'm willing to help too if you want to carve out a few classes for me. I
> think I can do it in my sleep now. FWIW, I will be out again this
> Thurs-Tues.
Re: Is o.a.m.df.mapred.* removeable?
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
+1 Having already done the port, I like the new APIs better than the
old. And I'm willing to help too if you want to carve out a few classes
for me. I think I can do it in my sleep now. FWIW, I will be out again
this Thurs-Tues.
On 6/14/10 2:25 PM, Sean Owen wrote:
> How odd. I'd still propose we move to the new APIs because the old
> ones are still deprecated, and the new ones are more sensible APIs I
> think, and I am willing to do most or all of the work.
>
> On Mon, Jun 14, 2010 at 10:02 PM, Grant Ingersoll<gs...@apache.org> wrote:
>
>> FWIW, someone at Buzzwords was telling me that Hadoop is undeprecating the old APIs in the next release. I find that even more confusing, but we might want to validate it and think about its implications.
>>
>
Re: Is o.a.m.df.mapred.* removeable?
Posted by Sean Owen <sr...@gmail.com>.
How odd. I'd still propose we move to the new APIs because the old
ones are still deprecated, and the new ones are more sensible APIs I
think, and I am willing to do most or all of the work.
On Mon, Jun 14, 2010 at 10:02 PM, Grant Ingersoll <gs...@apache.org> wrote:
> FWIW, someone at Buzzwords was telling me that Hadoop is undeprecating the old APIs in the next release. I find that even more confusing, but we might want to validate it and think about its implications.
Re: Is o.a.m.df.mapred.* removeable?
Posted by Grant Ingersoll <gs...@apache.org>.
FWIW, someone at Buzzwords was telling me that Hadoop is undeprecating the old APIs in the next release. I find that even more confusing, but we might want to validate it and think about its implications.
On Jun 14, 2010, at 1:43 PM, Sean Owen wrote:
> I'm about to make one of those big patches again. This time to push to
> update to use the new Hadoop APIs everywhere (purely updating the API
> calls).
>
> Am I right that o.a.m.df.mapreduce.* duplicates and supersedes
> o.a.m.df.mapred.*, the difference being that the former uses new APIs
> already? I have this from an earlier message but want to confirm.
>
> Sean
Re: Is o.a.m.df.mapred.* removeable?
Posted by deneche abdelhakim <ad...@gmail.com>.
one quick question ? I remember that the new API itself was using a
lot of deprecated stuff from the old API, is it still the case ?
On Tue, Jun 15, 2010 at 5:15 AM, deneche abdelhakim <ad...@gmail.com> wrote:
> yes, o.a.m.df.mapred.* uses the old APIs and o.a.m.df.mapreduce.* uses
> the new ones. In fact all my latest work have been done in
> o.a.m.df.mapreduce.* I just kept o.a.m.df.mapred.* for compatibility
> with the old APIs and in case we decide to not use the APIs (I
> remember we had quite a trouble with the new APIs).
>
> you can safely remove o.a.m.df.mapred.* and its related tests
>
> On Mon, Jun 14, 2010 at 6:43 PM, Sean Owen <sr...@gmail.com> wrote:
>> I'm about to make one of those big patches again. This time to push to
>> update to use the new Hadoop APIs everywhere (purely updating the API
>> calls).
>>
>> Am I right that o.a.m.df.mapreduce.* duplicates and supersedes
>> o.a.m.df.mapred.*, the difference being that the former uses new APIs
>> already? I have this from an earlier message but want to confirm.
>>
>> Sean
>>
>
Re: Is o.a.m.df.mapred.* removeable?
Posted by deneche abdelhakim <ad...@gmail.com>.
yes, o.a.m.df.mapred.* uses the old APIs and o.a.m.df.mapreduce.* uses
the new ones. In fact all my latest work have been done in
o.a.m.df.mapreduce.* I just kept o.a.m.df.mapred.* for compatibility
with the old APIs and in case we decide to not use the APIs (I
remember we had quite a trouble with the new APIs).
you can safely remove o.a.m.df.mapred.* and its related tests
On Mon, Jun 14, 2010 at 6:43 PM, Sean Owen <sr...@gmail.com> wrote:
> I'm about to make one of those big patches again. This time to push to
> update to use the new Hadoop APIs everywhere (purely updating the API
> calls).
>
> Am I right that o.a.m.df.mapreduce.* duplicates and supersedes
> o.a.m.df.mapred.*, the difference being that the former uses new APIs
> already? I have this from an earlier message but want to confirm.
>
> Sean
>