Re: Default hadoop version for 0.94?

[lars hofhansl <la...@apache.org>]

There appear to be significant performance advantages to using Hadoop 1.2.1.
I was just debugging why the VerifyReplication test takes 190s in 0.94 vs. 39s in HBase trunk.
After some debugging it turns out the difference is due to the different default Hadoop versions (0.94 uses 1.0.4, trunk uses 1.2.1).

I have not made the change, yet, but it seems nobody should really use 1.0.x anymore (if possible, of course).


-- Lars



________________________________
 From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; hbase-user <us...@hbase.apache.org> 
Sent: Thursday, October 17, 2013 10:39 PM
Subject: Re: Default hadoop version for 0.94?
 

+ User list

I am planning to upgrade 0.94's default Hadoop to 1.2.1.


Please speak up now if you think this is an issue.

Thanks.

-- Lars



________________________________

From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org> 
Sent: Thursday, October 17, 2013 7:10 PM
Subject: Re: Default hadoop version for 0.94?


Ok... I will make that so.
Of course the 1.0.x target will stay, just that the default of the sample tarballs will change to 1.2.1.

-- Lars



________________________________

From: Ted Yu <yu...@gmail.com>
To: "dev@hbase.apache.org" <de...@hbase.apache.org> 
Sent: Thursday, October 17, 2013 3:55 PM
Subject: Re: Default hadoop version for 0.94?


For hadoop-1.0 profile, currently version 1.0.4 is used.

Setting up a new build against hadoop 1.0.4 when the original build
switches to using 1.2.1 would be good.



On Thu, Oct 17, 2013 at 3:48 PM, Enis Söztutar <en...@gmail.com> wrote:

> +1 for hadoop 1.2. However, my only concern is that we should be able to
> still build and work with 1.0.3 and 1.1.x. Maybe we should create jenkins
> build for those?
>
>
> On Thu, Oct 17, 2013 at 3:39 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > Excellent. Thanks Ted.
> >
> >
> > 2013/10/17 Ted Yu <yu...@gmail.com>
> >
> > > bq.  Will that make anything faster to have 1.2 as the default version
> > > instead of 1.0?
> > >
> > > You can get to release notes from this thread:
> > > http://search-hadoop.com/m/pFao2kKHDm1
> > >
> > > A few examples:
> > >
> > > MAPREDUCE-5368 <https://issues.apache.org/jira/browse/MAPREDUCE-5368>.
> > > Major improvement reported by zhaoyunjiong and fixed by zhaoyunjiong
> > (mrv1)
> > > *Save memory by set capacity, load factor and concurrency level for
> > > ConcurrentHashMap in TaskInProgress*
> > > *
> > > *
> > > HADOOP-8971 <https://issues.apache.org/jira/browse/HADOOP-8971>. Major
> > > improvement reported by gopalv and fixed by gopalv (util)
> > > *Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static
> > data
> > > (HADOOP-8926)**
> > > *
> > >
> > >
> > > On Thu, Oct 17, 2013 at 11:58 AM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >
> > > > I think also we should be able to compile against v1 if required.
> Don't
> > > > think we should remove the related code yet. Might be on the roadmap,
> > but
> > > > most probably to soon to do that.  I'm not +1 because that will
> impact
> > me
> > > > ;) But can't be -1 (anyway, who cares, I'm not commiter ;)  )
> > > >
> > > > Just one question, that is this going to give us? Will that make
> > anything
> > > > faster to have 1.2 as the default version instead of 1.0?
> > > >
> > > >
> > > > 2013/10/17 Andrew Purtell <ap...@apache.org>
> > > >
> > > > > +1, if the current default version of 1 we build against can't even
> > be
> > > > > downloaded except from the archives, that's not good.
> > > > >
> > > > >
> > > > > On Thu, Oct 17, 2013 at 11:15 AM, Nick Dimiduk <ndimiduk@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > +1 from me as well. Keep on moving forward.
> > > > > >
> > > > > > Does this mean we can delete some old API compatibility code
> soon?
> > I
> > > <3
> > > > > > deleting code ;)
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 17, 2013 at 11:09 AM, lars hofhansl <
> larsh@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > So this is a resounding: "Maybe"? :)
> > > > > > >
> > > > > > >
> > > > > > > I think we should change the default. HBase should be built
> from
> > > > source
> > > > > > > for a specific version of Hadoop anyway.
> > > > > > >
> > > > > > > Ted said +1.
> > > > > > >
> > > > > > > Any other opinions?
> > > > > > >
> > > > > > >
> > > > > > > -- Lars
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > >  From: Jean-Marc Spaggiari <je...@spaggiari.org>
> > > > > > > To: dev <de...@hbase.apache.org>; lars hofhansl <
> larsh@apache.org>
> > > > > > > Sent: Wednesday, October 9, 2013 5:13 PM
> > > > > > > Subject: Re: Default hadoop version for 0.94?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > That's on my roadmap, but lacking some time for that ;) I
> bought
> > a
> > > > new
> > > > > > > cluster where I will install Hadoop 1.2.1 or 2.x and replicate
> my
> > > > > current
> > > > > > > cluster to it, but still need some time to do that :(
> > > > > > >
> > > > > > > But the issue is not just me. Me, I can deal with that. But
> maybe
> > > > > others
> > > > > > > might be in the same situation?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 2013/10/9 lars hofhansl <la...@apache.org>
> > > > > > >
> > > > > > > That's what I was trying to find out :)
> > > > > > > >You do not want to upgrade to Hadoop 1.2.1?
> > > > > > > >
> > > > > > > >
> > > > > > > >-- Lars
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >________________________________
> > > > > > > > From: Jean-Marc Spaggiari <je...@spaggiari.org>
> > > > > > > >To: dev <de...@hbase.apache.org>; lars hofhansl <
> larsh@apache.org
> > >
> > > > > > > >Sent: Wednesday, October 9, 2013 4:04 PM
> > > > > > > >Subject: Re: Default hadoop version for 0.94?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >Hum. I usually just deploy the .jar on my cluster (hadoop
> 1.0.3)
> > > > > without
> > > > > > > >rebuilding anything. Not sure if some others are not doing the
> > > same.
> > > > > > That
> > > > > > > >change will break the compatibility with previous version, no?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >2013/10/9 lars hofhansl <la...@apache.org>
> > > > > > > >
> > > > > > > >> Should we default the HBase 0.94 builds to Hadoop 1.2.x
> (1.2.1
> > > > > > > currently)?
> > > > > > > >> It's the current stable release of the Hadoop. Can't even
> > > download
> > > > > > 1.0.4
> > > > > > > >> anymore unless you navigate to the archive section.
> > > > > > > >>
> > > > > > > >> This would just apply to the sample packages in the download
> > > > section
> > > > > > for
> > > > > > > >> HBase. User should really build a targeted version of HBase
> to
> > > > their
> > > > > > > >> version of Hadoop anyway.
> > > > > > > >>
> > > > > > > >> Comments?
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> -- Lars
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > >    - Andy
> > > > >
> > > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein
> > > > > (via Tom White)
> > > > >
> > > >
> > >
> >
>