You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Paul Smith <ps...@aconex.com> on 2010/02/13 03:10:59 UTC
Moving to Maven (HBASE-2099)
I while ago I prepared a fairly lengthy email to this list about the potential of moving HBase to a Maven build system (as is discussed on HBASE-2099). On reflection, it was a) long, and b) a bit over the top. I think from a small smattering of discussions on IRC it appears there's at least a view that it might be worth kicking off a discussion, so the content of the original mail isn't needed.
So, lets talk! :)
I personally think Maven will assist as the HBase community to refactor the Hbase system into various components and should make your lives a lot easier. The built-in reporting and packaging elements make things easy if you follow conventions (I'm just proposing a few directory shuffles to follow these conventions to simplify things). I'm also recommending a slight change in direction with respect to the produced tar balls that are spit out.
What do people think? The IntelliJ built in support for Maven is outstanding, and the M2Eclipse plugin is excellent for Eclipse, so there's good support for it in the IDE. Kay Kay looks to have good Maven skills up his sleeve by the looks of his work on HBASE-2212 moving out the Lucene code to a github project (nice work!), so there should be good support around. Maven will help simplify the publishing of artifacts to the Maven repositories (particularly regular snapshots for anyone upstream that want's easy access to the latest without having to build from source).
HBASE-2099 has a lengthy comment trail on it, so may not be easily digestible. If you have any questions or concerns I'd really like to hear your thoughts. If you're unfamiliar with Maven I highly recommend the Maven: The Definitive Guide (free, see [1])
cheers,
Paul
[1] Maven: The Definitive Guide - http://www.sonatype.com/book/
(now split into 2 books, a by-example, and a reference guide).
Re: Moving to Maven (HBASE-2099)
Posted by Lars Francke <la...@gmail.com>.
> What do people think? The IntelliJ built in support for Maven is outstanding, and the M2Eclipse plugin is excellent for Eclipse, so there's good support for it in the IDE. Kay Kay looks to have good Maven skills up his sleeve by the looks of his work on HBASE-2212 moving out the Lucene code to a github project (nice work!), so there should be good support around. Maven will help simplify the publishing of artifacts to the Maven repositories (particularly regular snapshots for anyone upstream that want's easy access to the latest without having to build from source).
The IntelliJ support is indeed outstanding.
Thanks to Kay Kay and you for the work on this and the help with the
transition to Ivy which'll probably make the move to Maven easier.
I for one can't wait for a mavenized HBase: +1
Cheers,
Lars
Re: Moving to Maven (HBASE-2099)
Posted by Dan Washusen <da...@reactive.org>.
The first time Maven resolves a dependency it caches it in the
~/.m2/repostory/<groupId>/<atrifactId>/<version>/ directory. From this
point on, assuming you don't add or change dependencies, you can build
completely offline without any special flags.
If your project includes dependencies on snapshots it's a little different;
by default Maven will check if new snapshot versions of dependencies are
available every build. You can override this behavior by either building in
offline mode (-o) or by updating the snapshot policy. The snapshot policy
defines the frequency at which these checks are performed and can be
controlled per project and/or per developer. An individual developer on the
project can also override the project wide settings.
Another very handy feature is the ability to include javadoc and source for
project dependencies when generating Idea/Eclipse projects. For example, to
generate an Idea project with the ability to click through to dependency
code and view dependency javadoc I use (same offline/online rules apply):
mvn -DdownloadSources -DdownloadJavadocs idea:idea
Admittedly the black box nature of Maven means when things go wrong they
REALLY go wrong. However, I've been drinking the Maven Kool-Aid for a few
years now and I still seem to be breathing.
Drink the Kool-Aid... ;-P
Cheers,
Dan
On 14 February 2010 04:34, Andrew Purtell <ap...@apache.org> wrote:
> I suspect the problems I'd like to avoid we've already mostly taken on
> with Ivy. Primarily: The need to be connected to build and test. That is
> a real problem on a 15 hour flight. Domestic flights have options, but
> not the international ones. At least with Ivy I can edit out the resolve
> steps from the Ant buildfile as long as I ran 'ant test' before boarding.
> Edit, then rm -rf build/classes/* instead of 'ant clean', etc.
>
> Don't tell me I need to spend hours to figure out how to set up a local
> mirror of the public repo and then keep it up to date thereafter. There
> seem to be 5 large repositories referenced in ivysettings.xml. I doubt I
> have or can spare enough disk on the laptop and don't have the bandwidth
> to prune to just the packages that HBase needs and then update by hand
> whenever that changes or they change. If you tell me this, I have an
> impolite response for you.
>
> Clearly I'm unhappy with the definition of "progress" of some. Will
> Maven make me more or less unhappy? If Maven can support some totally
> disconnected operation from cache -- which Ivy cannot apparently -- then
> that would be awesome. If it makes the situation for me worse, I'll just
> do something else on those flights.
>
> - Andy
>
>
>
> ----- Original Message ----
> > From: Ryan Rawson <ry...@gmail.com>
> > To: hbase-dev@hadoop.apache.org
> > Sent: Sat, February 13, 2010 12:10:36 AM
> > Subject: Re: Moving to Maven (HBASE-2099)
> >
> > Would you mind elaborating more? At the moment, most people do not
> > build hbase, and the POM/jar/publishing thing is orthogonal - those
> > who wish to build their own projects with ivy and/or ant are free to
> > do so and not be impacted by our use of maven.
> >
> > We have ivy, but it doesnt integrate with our IDEs and is rather slow
> > to build and rebuild.
> >
> > On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
> > wrote:
> > > -1
> > >
> > > I think Maven is too complex and will lower the adoption of HBase by
> > > people today willing to build it.
> > >
> > > I would suggest using Ivy for dependency management as was done in
> Thrift.
> > >
> > > Mathias.
> > >
>
>
>
>
>
>
Re: Moving to Maven (HBASE-2099)
Posted by Stack <st...@duboce.net>.
This sounds broke Andrew -- like its not working as advertised. Make
an issue and the fellas w/ ivy-fu will fix it.
St.Ack
On Sat, Feb 13, 2010 at 9:48 AM, Andrew Purtell <ap...@apache.org> wrote:
> Even with -Doffline=true Ivy still wants to connect to resolve dependencies
> and fails the build if I'm disconnected. So I have to take the ivy rules
> out of the build file to do anything useful.
>
> Am I missing something?
>
> - Andy
>
>
>
> ----- Original Message ----
>> From: Ryan Rawson <ry...@gmail.com>
>> To: hbase-dev@hadoop.apache.org
>> Sent: Sat, February 13, 2010 9:39:51 AM
>> Subject: Re: Moving to Maven (HBASE-2099)
>>
>> Andrew... its not a problem, you run ant before You go and the deps are
>> automatically cached in a dot directory in your home.
>>
>> Woe is you if you forget to do this while connected: -(
>>
>> On Feb 13, 2010 9:35 AM, "Andrew Purtell" wrote:
>>
>> I suspect the problems I'd like to avoid we've already mostly taken on
>> with Ivy. Primarily: The need to be connected to build and test. That is
>> a real problem on a 15 hour flight. Domestic flights have options, but
>> not the international ones. At least with Ivy I can edit out the resolve
>> steps from the Ant buildfile as long as I ran 'ant test' before boarding.
>> Edit, then rm -rf build/classes/* instead of 'ant clean', etc.
>>
>> Don't tell me I need to spend hours to figure out how to set up a local
>> mirror of the public repo and then keep it up to date thereafter. There
>> seem to be 5 large repositories referenced in ivysettings.xml. I doubt I
>> have or can spare enough disk on the laptop and don't have the bandwidth
>> to prune to just the packages that HBase needs and then update by hand
>> whenever that changes or they change. If you tell me this, I have an
>> impolite response for you.
>>
>> Clearly I'm unhappy with the definition of "progress" of some. Will
>> Maven make me more or less unhappy? If Maven can support some totally
>> disconnected operation from cache -- which Ivy cannot apparently -- then
>> that would be awesome. If it makes the situation for me worse, I'll just
>> do something else on those flights.
>>
>> - Andy
>>
>>
>>
>>
>> ----- Original Message ----
>> > From: Ryan Rawson
>> > To: hbase-dev@hadoop.apach...
>>
>> > wrote:
>> > > -1
>> > >
>> > > I think Maven is too complex and will lower the adoption of HBase by
>> > > peo...
>
>
>
>
>
>
Re: Moving to Maven (HBASE-2099)
Posted by Andrew Purtell <ap...@apache.org>.
Even with -Doffline=true Ivy still wants to connect to resolve dependencies
and fails the build if I'm disconnected. So I have to take the ivy rules
out of the build file to do anything useful.
Am I missing something?
- Andy
----- Original Message ----
> From: Ryan Rawson <ry...@gmail.com>
> To: hbase-dev@hadoop.apache.org
> Sent: Sat, February 13, 2010 9:39:51 AM
> Subject: Re: Moving to Maven (HBASE-2099)
>
> Andrew... its not a problem, you run ant before You go and the deps are
> automatically cached in a dot directory in your home.
>
> Woe is you if you forget to do this while connected: -(
>
> On Feb 13, 2010 9:35 AM, "Andrew Purtell" wrote:
>
> I suspect the problems I'd like to avoid we've already mostly taken on
> with Ivy. Primarily: The need to be connected to build and test. That is
> a real problem on a 15 hour flight. Domestic flights have options, but
> not the international ones. At least with Ivy I can edit out the resolve
> steps from the Ant buildfile as long as I ran 'ant test' before boarding.
> Edit, then rm -rf build/classes/* instead of 'ant clean', etc.
>
> Don't tell me I need to spend hours to figure out how to set up a local
> mirror of the public repo and then keep it up to date thereafter. There
> seem to be 5 large repositories referenced in ivysettings.xml. I doubt I
> have or can spare enough disk on the laptop and don't have the bandwidth
> to prune to just the packages that HBase needs and then update by hand
> whenever that changes or they change. If you tell me this, I have an
> impolite response for you.
>
> Clearly I'm unhappy with the definition of "progress" of some. Will
> Maven make me more or less unhappy? If Maven can support some totally
> disconnected operation from cache -- which Ivy cannot apparently -- then
> that would be awesome. If it makes the situation for me worse, I'll just
> do something else on those flights.
>
> - Andy
>
>
>
>
> ----- Original Message ----
> > From: Ryan Rawson
> > To: hbase-dev@hadoop.apach...
>
> > wrote:
> > > -1
> > >
> > > I think Maven is too complex and will lower the adoption of HBase by
> > > peo...
Re: Moving to Maven (HBASE-2099)
Posted by Ryan Rawson <ry...@gmail.com>.
Andrew... its not a problem, you run ant before You go and the deps are
automatically cached in a dot directory in your home.
Woe is you if you forget to do this while connected: -(
On Feb 13, 2010 9:35 AM, "Andrew Purtell" <ap...@apache.org> wrote:
I suspect the problems I'd like to avoid we've already mostly taken on
with Ivy. Primarily: The need to be connected to build and test. That is
a real problem on a 15 hour flight. Domestic flights have options, but
not the international ones. At least with Ivy I can edit out the resolve
steps from the Ant buildfile as long as I ran 'ant test' before boarding.
Edit, then rm -rf build/classes/* instead of 'ant clean', etc.
Don't tell me I need to spend hours to figure out how to set up a local
mirror of the public repo and then keep it up to date thereafter. There
seem to be 5 large repositories referenced in ivysettings.xml. I doubt I
have or can spare enough disk on the laptop and don't have the bandwidth
to prune to just the packages that HBase needs and then update by hand
whenever that changes or they change. If you tell me this, I have an
impolite response for you.
Clearly I'm unhappy with the definition of "progress" of some. Will
Maven make me more or less unhappy? If Maven can support some totally
disconnected operation from cache -- which Ivy cannot apparently -- then
that would be awesome. If it makes the situation for me worse, I'll just
do something else on those flights.
- Andy
----- Original Message ----
> From: Ryan Rawson <ry...@gmail.com>
> To: hbase-dev@hadoop.apach...
> wrote:
> > -1
> >
> > I think Maven is too complex and will lower the adoption of HBase by
> > peo...
Re: Moving to Maven (HBASE-2099)
Posted by Andrew Purtell <ap...@apache.org>.
I suspect the problems I'd like to avoid we've already mostly taken on
with Ivy. Primarily: The need to be connected to build and test. That is
a real problem on a 15 hour flight. Domestic flights have options, but
not the international ones. At least with Ivy I can edit out the resolve
steps from the Ant buildfile as long as I ran 'ant test' before boarding.
Edit, then rm -rf build/classes/* instead of 'ant clean', etc.
Don't tell me I need to spend hours to figure out how to set up a local
mirror of the public repo and then keep it up to date thereafter. There
seem to be 5 large repositories referenced in ivysettings.xml. I doubt I
have or can spare enough disk on the laptop and don't have the bandwidth
to prune to just the packages that HBase needs and then update by hand
whenever that changes or they change. If you tell me this, I have an
impolite response for you.
Clearly I'm unhappy with the definition of "progress" of some. Will
Maven make me more or less unhappy? If Maven can support some totally
disconnected operation from cache -- which Ivy cannot apparently -- then
that would be awesome. If it makes the situation for me worse, I'll just
do something else on those flights.
- Andy
----- Original Message ----
> From: Ryan Rawson <ry...@gmail.com>
> To: hbase-dev@hadoop.apache.org
> Sent: Sat, February 13, 2010 12:10:36 AM
> Subject: Re: Moving to Maven (HBASE-2099)
>
> Would you mind elaborating more? At the moment, most people do not
> build hbase, and the POM/jar/publishing thing is orthogonal - those
> who wish to build their own projects with ivy and/or ant are free to
> do so and not be impacted by our use of maven.
>
> We have ivy, but it doesnt integrate with our IDEs and is rather slow
> to build and rebuild.
>
> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
> wrote:
> > -1
> >
> > I think Maven is too complex and will lower the adoption of HBase by
> > people today willing to build it.
> >
> > I would suggest using Ivy for dependency management as was done in Thrift.
> >
> > Mathias.
> >
Re: Moving to Maven (HBASE-2099)
Posted by Mathias Herberts <ma...@gmail.com>.
On Sat, Feb 13, 2010 at 09:10, Ryan Rawson <ry...@gmail.com> wrote:
> Would you mind elaborating more? At the moment, most people do not
> build hbase, and the POM/jar/publishing thing is orthogonal - those
> who wish to build their own projects with ivy and/or ant are free to
> do so and not be impacted by our use of maven.
Sure,
I've had in the past several bad experiences with project using maven,
especially with inter project dependencies. I guess what Kay has been
saying about Hbase's build being pretty straight forward won't make
people bump into this kind of knigtmare.
As for the slowness I don't think it is due to Ivy per se, and from my
experience, neither maven nor Ivy show any really speed difference.
I withdraw my -1 and replace it with a 0.
Mathias
Re: Moving to Maven (HBASE-2099)
Posted by Kay Kay <ka...@gmail.com>.
Given that both ivy-maven-stuff address the issue of dependency
management, I can broadly think of 2 reasons (the producer and the
consumer) and other advantages coming under them one way or another.
* The maintainer of the project would be responsible for handling
dependencies and upgrading them on a need basis.
Just like any other encapsulation we talk about, the children
dependencies (level 2 of the dependency tree) of the primary
dependencies that we are concerned with, are entirely hidden from us
since that would be the job of the corresponding package owner to
identify and list the same (pom.xml / ivy.xml , as appropriate) , and
not that of the user of the package.
With appropriate test suites in place, it would be easy to flip /
test-drive a new version before reverting back / upgrading , especially
when a project publishes more than 1 artifact.
* As a consumer of the project, if a maintainer puts the artifacts
along with the dependencies, that makes it easy to assemble the blocks
with only the primary dependencies listed ( and not worrying about other
libraries, that might be needed , encountering nasty ClassNotFoundErrors
in runtime).
I agree that both of them sound very theoretical and ideal . The case
where it works straight-forward is when you are setting up a project -
you can get up and running with minimal bullet points that you know as a
dependency of your project , without trying to gather the entire list
needed for the same, behind the scenes. Since with mvn / ivy , that is
already defined by the package maintainer and best to use that.
This might not make sense for hadoop, but for a much larger codebase, as
consumers of the same - it would be very useful to have these blocks
'blessed' by the original contributors / maintainers - so they can keep
up / test drive new versions before actually making a decision one way
or another, as opposed to ( remove all old versions and download and add
new versions in ./lib directory ).
Recently , we had an upgradation of the thrift library and during the
process -there were some discussions about some of the guts of thrift
code, using commons-lang for a hashcode implementation (or s.th ,
similar to that). While that was definitely informative, as users of the
library - it is something that should have been transparent , if the
process was already existing as opposed to getting into the guts of it.
On the other end of the spectrum - currently , if anybody is planning to
get started with just the client framework of HBase to communicate with
the eco-system ( zk , master, r.s etc. ) , they have to get every other
dependency listed at hbase (not every body has the time/ resources to
play around with sources and figure out the subset and restrict it) .
Assuming the server setup is all complete and clients living in a
different machine - all that is needed is a scaled down version of
client library (and not the giant list), that knows the ipc semantics ,
without worrying about the server internals. So - on the publication
side- once ivy/maven-ization is complete , hbase can start publishing
different artifacts that could be used , depending on the needs.
A great candidate for such a case would be mahout project, that uses
hbase for one of its algorithm implementations, and there is no
necessity for a giant load of hbase on them, say.
Specifically for hdfs (and hadoop projects, in general) - I can
definitely see the frustration coming from , due to the necessity to
keep up with snapshots that is in a state of flux / transformation.
That in turn slows down the build time. Ironically - that would be the
right way to keep up with the dependencies and as consumers of other
projects - we can be less aggressive about how frequently we want to
keep up with upstream projects, while having an easy option to try out
new ones as they become available.
As we see - it is just another eco-system, that can thrive well - when
everybody plays by the rules. Bad / inconsistent pom-s, ivy-s definitely
do exist , as I discovered (on some of the projects of hadoop-xyz).
While that can be extremely frustrating - assuming the community is open
to receive patches / address them - will help thrive the eco-system
better. Viewing the dependency graph of a given project with other
stakeholders in the same room, (applicable within a 'shop' as opposed to
'oss' projects ) , will bring forth 'ball-of-mud' / 'code duplication'
design patterns by using some graph theory 101 principles and give an
idea , when people are starting to refactor existing code. ( Ideally it
should be a dependency tree, but for all practical purposes - it becomes
a dependency graph for a variety of reasons, some of them non-technical
as anybody can guess ! ).
As Mathias had pointed out - while maven / ivy solve the dependency
management problem, maven's rigid rules sometime make the build process
go crazy (especially when we try to transfer from non-dependency
management world with custom scripts / unconventional 'target'
lifecycles ). I am +1-on the maven part of HBase since I do not see the
need for such a process , if we keep the scope of hbase-core restricted
to what it does today, with other plugins to hbase, appearing as
separate libraries / apps as opposed to overloading the codebase with
more responsibilities. The detailed comparison between maven and ivy
warrant a separate post altogether, so am not going to continue on that.
I have listed them , from my own experiences and do not reflect the
official stance behind 'ivy'-ization of hadoop / even hbase , for that
matter. In reality - for the longevity of code base, such 'blessed'
eco-system helps everyone in it.
--
K K.
On 02/13/2010 01:51 AM, Dhruba Borthakur wrote:
> My personal experience is that the ivy-maven-stuff introduced into the
> Hadoop build system has tremendously slowed down the Hadoop build process. I
> am sure that this disadvantage is offet-ed by some advantages that I am not
> aware of. If you could educate me on the top two advantages that accrued to
> Hadoop after moving to the new build process, that would be awesome.
>
> thanks a bunch,
> dhruba
>
>
> On Sat, Feb 13, 2010 at 1:44 AM, Kay Kay<ka...@gmail.com> wrote:
>
>
>> On 02/13/2010 01:29 AM, Dhruba Borthakur wrote:
>>
>>
>>> From what I understand, the slowness of 'ivy' can be reduced if you can
>>> fetch dependent jars from local ivy server, isn't it?
>>>
>>>
>>>
>> The problem discussed is an artifact of hbase , trying to keep up with the
>> most recent snapshots of hadoop-core/ hdfs / mapred and hence the ivy
>> resolution is expensive that every time it hits the mvn repository to check
>> for the latest snapshot , if any. So the slowness is due to the necessity
>> to keep up with the dependencies to identify issues early in the cycle.
>> Specifically this can be attributed to the - changing="true" in all the
>> ivy.xml-s in hbase, for hadoop artifacts . I am looking to making it a
>> configurable option to avoid expensive build time.
>>
>>
>> This will not be an issue if this were a hbase release, depending on other
>> releases of hadoop-core / mapred / common etc.
>> Both ivy and maven does cache the artifacts locally making the roundtrip
>> redundant (except for the first time, of course), so this should not be an
>> issue for people trying to build the release from sources, since it should
>> be moot by then.
>>
>>
>>
>>
>>
>> thanks,
>>
>>> dhruba
>>>
>>> On Sat, Feb 13, 2010 at 12:25 AM, Kay Kay<ka...@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>>
>>>> Mathias -
>>>> I have been using Ivy / Maven , interchangeably in different projects
>>>> for
>>>> the build management. Both of them clearly have their strong points and
>>>> drawbacks. Ivy fits great for thrift because of the nature of tasks ,
>>>> involved using some external command-line (thrift generators) etc. As I
>>>> mentioned before - HBase does not have such cross maven goals / between
>>>> the
>>>> hairs as the build lifecycle is pretty straight-forward.
>>>> In any case - the intention is to get to publish HBase artifacts and
>>>> maintain a smaller core and encouraging contribs. from the artifacts as
>>>> opposed to getting into the codebase.
>>>> Once there are HBase artifacts published , the contrib / plugins for the
>>>> same would be free to use ivy (with m2compatible="true") / maven as
>>>> appropriate.
>>>>
>>>> Ryan -
>>>> The slowness is attributed to the 'changing="true" ' in ivy.xml-s for
>>>> all
>>>> the hadoop-common / -hdfs / -mapreduce snapshots that we are using. I am
>>>> facing similar 'slowness' with other mvn hadoop (snapshot) dependencies
>>>> as
>>>> well. In retrospective, that should have been made a configurable flag in
>>>> libraries.properties , to ease things. Hopefully that is sorted out soon.
>>>>
>>>>
>>>>
>>>>
>>>> On 02/13/2010 12:10 AM, Ryan Rawson wrote:
>>>>
>>>>
>>>>
>>>>
>>>>> Would you mind elaborating more? At the moment, most people do not
>>>>> build hbase, and the POM/jar/publishing thing is orthogonal - those
>>>>> who wish to build their own projects with ivy and/or ant are free to
>>>>> do so and not be impacted by our use of maven.
>>>>>
>>>>> We have ivy, but it doesnt integrate with our IDEs and is rather slow
>>>>> to build and rebuild.
>>>>>
>>>>> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
>>>>> <ma...@gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> -1
>>>>>>
>>>>>> I think Maven is too complex and will lower the adoption of HBase by
>>>>>> people today willing to build it.
>>>>>>
>>>>>> I would suggest using Ivy for dependency management as was done in
>>>>>> Thrift.
>>>>>>
>>>>>> Mathias.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Moving to Maven (HBASE-2099)
Posted by Stack <st...@duboce.net>.
On Sat, Feb 13, 2010 at 1:51 AM, Dhruba Borthakur <dh...@gmail.com> wrote:
> My personal experience is that the ivy-maven-stuff introduced into the
> Hadoop build system has tremendously slowed down the Hadoop build process. I
> am sure that this disadvantage is offet-ed by some advantages that I am not
> aware of. If you could educate me on the top two advantages that accrued to
> Hadoop after moving to the new build process, that would be awesome.
>
I'll try and answer your question Dhruba.
The hadoop build system change was done over in HADOOP-3305. The
justification for the change was, to quote Steve, "To let people
downstream build/test with hadoop, using Apache Ivy or Apache Maven2
to pull it down, hadoop-core needs to be published to the apache
repository with a .pom file that lists its mandatory dependencies."
Ivy was introduced to facilitate the above.
I don't see too much discussion of it up on hadoop lists (My search
may be wonky). I see Ashish asking back in December of 2008 about how
to manage dependencies between subprojects and hadoop core and the ivy
being suggested because it doesn't replace ant, whereas maven does.
So, it would seem there was only one advantage/improvement brought by
the change in the hadoop build system: hadoop artifacts are now
available in maven repositories -- I don't think they are published
other than to the apache maven repository (ivy knows how to deal with
maven repos) -- with a pom that describes their dependencies so that a
maven/ivy build downstream can pull and integrate hadoop artifacts.
St.Ack
> thanks a bunch,
> dhruba
>
>
> On Sat, Feb 13, 2010 at 1:44 AM, Kay Kay <ka...@gmail.com> wrote:
>
>> On 02/13/2010 01:29 AM, Dhruba Borthakur wrote:
>>
>>> From what I understand, the slowness of 'ivy' can be reduced if you can
>>> fetch dependent jars from local ivy server, isn't it?
>>>
>>>
>> The problem discussed is an artifact of hbase , trying to keep up with the
>> most recent snapshots of hadoop-core/ hdfs / mapred and hence the ivy
>> resolution is expensive that every time it hits the mvn repository to check
>> for the latest snapshot , if any. So the slowness is due to the necessity
>> to keep up with the dependencies to identify issues early in the cycle.
>> Specifically this can be attributed to the - changing="true" in all the
>> ivy.xml-s in hbase, for hadoop artifacts . I am looking to making it a
>> configurable option to avoid expensive build time.
>>
>>
>> This will not be an issue if this were a hbase release, depending on other
>> releases of hadoop-core / mapred / common etc.
>> Both ivy and maven does cache the artifacts locally making the roundtrip
>> redundant (except for the first time, of course), so this should not be an
>> issue for people trying to build the release from sources, since it should
>> be moot by then.
>>
>>
>>
>>
>>
>> thanks,
>>> dhruba
>>>
>>> On Sat, Feb 13, 2010 at 12:25 AM, Kay Kay<ka...@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>>> Mathias -
>>>> I have been using Ivy / Maven , interchangeably in different projects
>>>> for
>>>> the build management. Both of them clearly have their strong points and
>>>> drawbacks. Ivy fits great for thrift because of the nature of tasks ,
>>>> involved using some external command-line (thrift generators) etc. As I
>>>> mentioned before - HBase does not have such cross maven goals / between
>>>> the
>>>> hairs as the build lifecycle is pretty straight-forward.
>>>> In any case - the intention is to get to publish HBase artifacts and
>>>> maintain a smaller core and encouraging contribs. from the artifacts as
>>>> opposed to getting into the codebase.
>>>> Once there are HBase artifacts published , the contrib / plugins for the
>>>> same would be free to use ivy (with m2compatible="true") / maven as
>>>> appropriate.
>>>>
>>>> Ryan -
>>>> The slowness is attributed to the 'changing="true" ' in ivy.xml-s for
>>>> all
>>>> the hadoop-common / -hdfs / -mapreduce snapshots that we are using. I am
>>>> facing similar 'slowness' with other mvn hadoop (snapshot) dependencies
>>>> as
>>>> well. In retrospective, that should have been made a configurable flag in
>>>> libraries.properties , to ease things. Hopefully that is sorted out soon.
>>>>
>>>>
>>>>
>>>>
>>>> On 02/13/2010 12:10 AM, Ryan Rawson wrote:
>>>>
>>>>
>>>>
>>>>> Would you mind elaborating more? At the moment, most people do not
>>>>> build hbase, and the POM/jar/publishing thing is orthogonal - those
>>>>> who wish to build their own projects with ivy and/or ant are free to
>>>>> do so and not be impacted by our use of maven.
>>>>>
>>>>> We have ivy, but it doesnt integrate with our IDEs and is rather slow
>>>>> to build and rebuild.
>>>>>
>>>>> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
>>>>> <ma...@gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> -1
>>>>>>
>>>>>> I think Maven is too complex and will lower the adoption of HBase by
>>>>>> people today willing to build it.
>>>>>>
>>>>>> I would suggest using Ivy for dependency management as was done in
>>>>>> Thrift.
>>>>>>
>>>>>> Mathias.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
Re: Moving to Maven (HBASE-2099)
Posted by Dan Washusen <da...@reactive.org>.
I stand corrected. Maven checks for snapshot updates once a day (according
to that link below)...
On 14 February 2010 12:26, Paul Smith <ps...@aconex.com> wrote:
>
> On 13/02/2010, at 9:11 PM, Dan Washusen wrote:
>
> > Maven provides a "-o" argument and Ivy provides a "-Doffline=true"
> > property. I haven't used the Ivy variant but if you're using SNAPSHOT
> > dependencies it makes the build a lot quicker. I usually do an
> infrequent
> > online build then do the rest in offline mode...
> >
> > Maven also provides a "snapshot policy" that can be used to define the
> > interval between snapshot checks. For example, "always", "daily",
> > "interval:<in minutes>" or "never"...
> >
> http://maven.apache.org/ref/2.0.8/maven-settings/settings.html#class_snapshots
> >
>
> As I understand it, I think the Maven default setting is no more than once
> or twice a day to check for snapshot updates. I wonder if the current ivy
> setting is just checking too often at the moment? (perhaps configurable).
>
> 'mvn -U' will force an update, I think '-o' for offline is a larger hammer
> that shouldn't be necessary.
>
> Paul
Re: Moving to Maven (HBASE-2099)
Posted by Paul Smith <ps...@aconex.com>.
On 13/02/2010, at 9:11 PM, Dan Washusen wrote:
> Maven provides a "-o" argument and Ivy provides a "-Doffline=true"
> property. I haven't used the Ivy variant but if you're using SNAPSHOT
> dependencies it makes the build a lot quicker. I usually do an infrequent
> online build then do the rest in offline mode...
>
> Maven also provides a "snapshot policy" that can be used to define the
> interval between snapshot checks. For example, "always", "daily",
> "interval:<in minutes>" or "never"...
> http://maven.apache.org/ref/2.0.8/maven-settings/settings.html#class_snapshots
>
As I understand it, I think the Maven default setting is no more than once or twice a day to check for snapshot updates. I wonder if the current ivy setting is just checking too often at the moment? (perhaps configurable).
'mvn -U' will force an update, I think '-o' for offline is a larger hammer that shouldn't be necessary.
Paul
Re: Moving to Maven (HBASE-2099)
Posted by Andrew Purtell <ap...@apache.org>.
-Doffline=true still wants to connect to resolve dependencies, so I think
it is a mis-named option.
----- Original Message ----
> From: Dan Washusen <da...@reactive.org>
> To: hbase-dev@hadoop.apache.org
> Sent: Sat, February 13, 2010 2:11:13 AM
> Subject: Re: Moving to Maven (HBASE-2099)
>
> Maven provides a "-o" argument and Ivy provides a "-Doffline=true"
> property. I haven't used the Ivy variant but if you're using SNAPSHOT
> dependencies it makes the build a lot quicker. I usually do an infrequent
> online build then do the rest in offline mode...
>
> Maven also provides a "snapshot policy" that can be used to define the
> interval between snapshot checks. For example, "always", "daily",
> "interval:" or "never"...
> http://maven.apache.org/ref/2.0.8/maven-settings/settings.html#class_snapshots
>
> Cheers,
> Dan
>
> On 13 February 2010 20:51, Dhruba Borthakur wrote:
>
> > My personal experience is that the ivy-maven-stuff introduced into the
> > Hadoop build system has tremendously slowed down the Hadoop build process.
> > I
> > am sure that this disadvantage is offet-ed by some advantages that I am not
> > aware of. If you could educate me on the top two advantages that accrued to
> > Hadoop after moving to the new build process, that would be awesome.
> >
> > thanks a bunch,
> > dhruba
> >
> >
> > On Sat, Feb 13, 2010 at 1:44 AM, Kay Kay wrote:
> >
> > > On 02/13/2010 01:29 AM, Dhruba Borthakur wrote:
> > >
> > >> From what I understand, the slowness of 'ivy' can be reduced if you can
> > >> fetch dependent jars from local ivy server, isn't it?
> > >>
> > >>
> > > The problem discussed is an artifact of hbase , trying to keep up with
> > the
> > > most recent snapshots of hadoop-core/ hdfs / mapred and hence the ivy
> > > resolution is expensive that every time it hits the mvn repository to
> > check
> > > for the latest snapshot , if any. So the slowness is due to the
> > necessity
> > > to keep up with the dependencies to identify issues early in the cycle.
> > > Specifically this can be attributed to the - changing="true" in all the
> > > ivy.xml-s in hbase, for hadoop artifacts . I am looking to making it a
> > > configurable option to avoid expensive build time.
> > >
> > >
> > > This will not be an issue if this were a hbase release, depending on
> > other
> > > releases of hadoop-core / mapred / common etc.
> > > Both ivy and maven does cache the artifacts locally making the roundtrip
> > > redundant (except for the first time, of course), so this should not be
> > an
> > > issue for people trying to build the release from sources, since it
> > should
> > > be moot by then.
> > >
> > >
> > >
> > >
> > >
> > > thanks,
> > >> dhruba
> > >>
> > >> On Sat, Feb 13, 2010 at 12:25 AM, Kay Kay
> > >> wrote:
> > >>
> > >>
> > >>
> > >>> Mathias -
> > >>> I have been using Ivy / Maven , interchangeably in different projects
> > >>> for
> > >>> the build management. Both of them clearly have their strong points
> > and
> > >>> drawbacks. Ivy fits great for thrift because of the nature of tasks ,
> > >>> involved using some external command-line (thrift generators) etc. As
> > I
> > >>> mentioned before - HBase does not have such cross maven goals / between
> > >>> the
> > >>> hairs as the build lifecycle is pretty straight-forward.
> > >>> In any case - the intention is to get to publish HBase artifacts and
> > >>> maintain a smaller core and encouraging contribs. from the artifacts as
> > >>> opposed to getting into the codebase.
> > >>> Once there are HBase artifacts published , the contrib / plugins for
> > the
> > >>> same would be free to use ivy (with m2compatible="true") / maven as
> > >>> appropriate.
> > >>>
> > >>> Ryan -
> > >>> The slowness is attributed to the 'changing="true" ' in ivy.xml-s for
> > >>> all
> > >>> the hadoop-common / -hdfs / -mapreduce snapshots that we are using. I
> > am
> > >>> facing similar 'slowness' with other mvn hadoop (snapshot) dependencies
> > >>> as
> > >>> well. In retrospective, that should have been made a configurable flag
> > in
> > >>> libraries.properties , to ease things. Hopefully that is sorted out
> > soon.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On 02/13/2010 12:10 AM, Ryan Rawson wrote:
> > >>>
> > >>>
> > >>>
> > >>>> Would you mind elaborating more? At the moment, most people do not
> > >>>> build hbase, and the POM/jar/publishing thing is orthogonal - those
> > >>>> who wish to build their own projects with ivy and/or ant are free to
> > >>>> do so and not be impacted by our use of maven.
> > >>>>
> > >>>> We have ivy, but it doesnt integrate with our IDEs and is rather slow
> > >>>> to build and rebuild.
> > >>>>
> > >>>> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
> > >>>> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>> -1
> > >>>>>
> > >>>>> I think Maven is too complex and will lower the adoption of HBase by
> > >>>>> people today willing to build it.
> > >>>>>
> > >>>>> I would suggest using Ivy for dependency management as was done in
> > >>>>> Thrift.
> > >>>>>
> > >>>>> Mathias.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >
> > >
> >
> >
> > --
> > Connect to me at http://www.facebook.com/dhruba
> >
Re: Moving to Maven (HBASE-2099)
Posted by Dan Washusen <da...@reactive.org>.
Maven provides a "-o" argument and Ivy provides a "-Doffline=true"
property. I haven't used the Ivy variant but if you're using SNAPSHOT
dependencies it makes the build a lot quicker. I usually do an infrequent
online build then do the rest in offline mode...
Maven also provides a "snapshot policy" that can be used to define the
interval between snapshot checks. For example, "always", "daily",
"interval:<in minutes>" or "never"...
http://maven.apache.org/ref/2.0.8/maven-settings/settings.html#class_snapshots
Cheers,
Dan
On 13 February 2010 20:51, Dhruba Borthakur <dh...@gmail.com> wrote:
> My personal experience is that the ivy-maven-stuff introduced into the
> Hadoop build system has tremendously slowed down the Hadoop build process.
> I
> am sure that this disadvantage is offet-ed by some advantages that I am not
> aware of. If you could educate me on the top two advantages that accrued to
> Hadoop after moving to the new build process, that would be awesome.
>
> thanks a bunch,
> dhruba
>
>
> On Sat, Feb 13, 2010 at 1:44 AM, Kay Kay <ka...@gmail.com> wrote:
>
> > On 02/13/2010 01:29 AM, Dhruba Borthakur wrote:
> >
> >> From what I understand, the slowness of 'ivy' can be reduced if you can
> >> fetch dependent jars from local ivy server, isn't it?
> >>
> >>
> > The problem discussed is an artifact of hbase , trying to keep up with
> the
> > most recent snapshots of hadoop-core/ hdfs / mapred and hence the ivy
> > resolution is expensive that every time it hits the mvn repository to
> check
> > for the latest snapshot , if any. So the slowness is due to the
> necessity
> > to keep up with the dependencies to identify issues early in the cycle.
> > Specifically this can be attributed to the - changing="true" in all the
> > ivy.xml-s in hbase, for hadoop artifacts . I am looking to making it a
> > configurable option to avoid expensive build time.
> >
> >
> > This will not be an issue if this were a hbase release, depending on
> other
> > releases of hadoop-core / mapred / common etc.
> > Both ivy and maven does cache the artifacts locally making the roundtrip
> > redundant (except for the first time, of course), so this should not be
> an
> > issue for people trying to build the release from sources, since it
> should
> > be moot by then.
> >
> >
> >
> >
> >
> > thanks,
> >> dhruba
> >>
> >> On Sat, Feb 13, 2010 at 12:25 AM, Kay Kay<ka...@gmail.com>
> >> wrote:
> >>
> >>
> >>
> >>> Mathias -
> >>> I have been using Ivy / Maven , interchangeably in different projects
> >>> for
> >>> the build management. Both of them clearly have their strong points
> and
> >>> drawbacks. Ivy fits great for thrift because of the nature of tasks ,
> >>> involved using some external command-line (thrift generators) etc. As
> I
> >>> mentioned before - HBase does not have such cross maven goals / between
> >>> the
> >>> hairs as the build lifecycle is pretty straight-forward.
> >>> In any case - the intention is to get to publish HBase artifacts and
> >>> maintain a smaller core and encouraging contribs. from the artifacts as
> >>> opposed to getting into the codebase.
> >>> Once there are HBase artifacts published , the contrib / plugins for
> the
> >>> same would be free to use ivy (with m2compatible="true") / maven as
> >>> appropriate.
> >>>
> >>> Ryan -
> >>> The slowness is attributed to the 'changing="true" ' in ivy.xml-s for
> >>> all
> >>> the hadoop-common / -hdfs / -mapreduce snapshots that we are using. I
> am
> >>> facing similar 'slowness' with other mvn hadoop (snapshot) dependencies
> >>> as
> >>> well. In retrospective, that should have been made a configurable flag
> in
> >>> libraries.properties , to ease things. Hopefully that is sorted out
> soon.
> >>>
> >>>
> >>>
> >>>
> >>> On 02/13/2010 12:10 AM, Ryan Rawson wrote:
> >>>
> >>>
> >>>
> >>>> Would you mind elaborating more? At the moment, most people do not
> >>>> build hbase, and the POM/jar/publishing thing is orthogonal - those
> >>>> who wish to build their own projects with ivy and/or ant are free to
> >>>> do so and not be impacted by our use of maven.
> >>>>
> >>>> We have ivy, but it doesnt integrate with our IDEs and is rather slow
> >>>> to build and rebuild.
> >>>>
> >>>> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
> >>>> <ma...@gmail.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> -1
> >>>>>
> >>>>> I think Maven is too complex and will lower the adoption of HBase by
> >>>>> people today willing to build it.
> >>>>>
> >>>>> I would suggest using Ivy for dependency management as was done in
> >>>>> Thrift.
> >>>>>
> >>>>> Mathias.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >
> >
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
Re: Moving to Maven (HBASE-2099)
Posted by Dhruba Borthakur <dh...@gmail.com>.
My personal experience is that the ivy-maven-stuff introduced into the
Hadoop build system has tremendously slowed down the Hadoop build process. I
am sure that this disadvantage is offet-ed by some advantages that I am not
aware of. If you could educate me on the top two advantages that accrued to
Hadoop after moving to the new build process, that would be awesome.
thanks a bunch,
dhruba
On Sat, Feb 13, 2010 at 1:44 AM, Kay Kay <ka...@gmail.com> wrote:
> On 02/13/2010 01:29 AM, Dhruba Borthakur wrote:
>
>> From what I understand, the slowness of 'ivy' can be reduced if you can
>> fetch dependent jars from local ivy server, isn't it?
>>
>>
> The problem discussed is an artifact of hbase , trying to keep up with the
> most recent snapshots of hadoop-core/ hdfs / mapred and hence the ivy
> resolution is expensive that every time it hits the mvn repository to check
> for the latest snapshot , if any. So the slowness is due to the necessity
> to keep up with the dependencies to identify issues early in the cycle.
> Specifically this can be attributed to the - changing="true" in all the
> ivy.xml-s in hbase, for hadoop artifacts . I am looking to making it a
> configurable option to avoid expensive build time.
>
>
> This will not be an issue if this were a hbase release, depending on other
> releases of hadoop-core / mapred / common etc.
> Both ivy and maven does cache the artifacts locally making the roundtrip
> redundant (except for the first time, of course), so this should not be an
> issue for people trying to build the release from sources, since it should
> be moot by then.
>
>
>
>
>
> thanks,
>> dhruba
>>
>> On Sat, Feb 13, 2010 at 12:25 AM, Kay Kay<ka...@gmail.com>
>> wrote:
>>
>>
>>
>>> Mathias -
>>> I have been using Ivy / Maven , interchangeably in different projects
>>> for
>>> the build management. Both of them clearly have their strong points and
>>> drawbacks. Ivy fits great for thrift because of the nature of tasks ,
>>> involved using some external command-line (thrift generators) etc. As I
>>> mentioned before - HBase does not have such cross maven goals / between
>>> the
>>> hairs as the build lifecycle is pretty straight-forward.
>>> In any case - the intention is to get to publish HBase artifacts and
>>> maintain a smaller core and encouraging contribs. from the artifacts as
>>> opposed to getting into the codebase.
>>> Once there are HBase artifacts published , the contrib / plugins for the
>>> same would be free to use ivy (with m2compatible="true") / maven as
>>> appropriate.
>>>
>>> Ryan -
>>> The slowness is attributed to the 'changing="true" ' in ivy.xml-s for
>>> all
>>> the hadoop-common / -hdfs / -mapreduce snapshots that we are using. I am
>>> facing similar 'slowness' with other mvn hadoop (snapshot) dependencies
>>> as
>>> well. In retrospective, that should have been made a configurable flag in
>>> libraries.properties , to ease things. Hopefully that is sorted out soon.
>>>
>>>
>>>
>>>
>>> On 02/13/2010 12:10 AM, Ryan Rawson wrote:
>>>
>>>
>>>
>>>> Would you mind elaborating more? At the moment, most people do not
>>>> build hbase, and the POM/jar/publishing thing is orthogonal - those
>>>> who wish to build their own projects with ivy and/or ant are free to
>>>> do so and not be impacted by our use of maven.
>>>>
>>>> We have ivy, but it doesnt integrate with our IDEs and is rather slow
>>>> to build and rebuild.
>>>>
>>>> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
>>>> <ma...@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>> -1
>>>>>
>>>>> I think Maven is too complex and will lower the adoption of HBase by
>>>>> people today willing to build it.
>>>>>
>>>>> I would suggest using Ivy for dependency management as was done in
>>>>> Thrift.
>>>>>
>>>>> Mathias.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
--
Connect to me at http://www.facebook.com/dhruba
Re: Moving to Maven (HBASE-2099)
Posted by Kay Kay <ka...@gmail.com>.
On 02/13/2010 01:29 AM, Dhruba Borthakur wrote:
> From what I understand, the slowness of 'ivy' can be reduced if you can
> fetch dependent jars from local ivy server, isn't it?
>
The problem discussed is an artifact of hbase , trying to keep up with
the most recent snapshots of hadoop-core/ hdfs / mapred and hence the
ivy resolution is expensive that every time it hits the mvn repository
to check for the latest snapshot , if any. So the slowness is due to
the necessity to keep up with the dependencies to identify issues early
in the cycle. Specifically this can be attributed to the -
changing="true" in all the ivy.xml-s in hbase, for hadoop artifacts . I
am looking to making it a configurable option to avoid expensive build
time.
This will not be an issue if this were a hbase release, depending on
other releases of hadoop-core / mapred / common etc.
Both ivy and maven does cache the artifacts locally making the roundtrip
redundant (except for the first time, of course), so this should not be
an issue for people trying to build the release from sources, since it
should be moot by then.
> thanks,
> dhruba
>
> On Sat, Feb 13, 2010 at 12:25 AM, Kay Kay<ka...@gmail.com> wrote:
>
>
>> Mathias -
>> I have been using Ivy / Maven , interchangeably in different projects for
>> the build management. Both of them clearly have their strong points and
>> drawbacks. Ivy fits great for thrift because of the nature of tasks ,
>> involved using some external command-line (thrift generators) etc. As I
>> mentioned before - HBase does not have such cross maven goals / between the
>> hairs as the build lifecycle is pretty straight-forward.
>> In any case - the intention is to get to publish HBase artifacts and
>> maintain a smaller core and encouraging contribs. from the artifacts as
>> opposed to getting into the codebase.
>> Once there are HBase artifacts published , the contrib / plugins for the
>> same would be free to use ivy (with m2compatible="true") / maven as
>> appropriate.
>>
>> Ryan -
>> The slowness is attributed to the 'changing="true" ' in ivy.xml-s for all
>> the hadoop-common / -hdfs / -mapreduce snapshots that we are using. I am
>> facing similar 'slowness' with other mvn hadoop (snapshot) dependencies as
>> well. In retrospective, that should have been made a configurable flag in
>> libraries.properties , to ease things. Hopefully that is sorted out soon.
>>
>>
>>
>>
>> On 02/13/2010 12:10 AM, Ryan Rawson wrote:
>>
>>
>>> Would you mind elaborating more? At the moment, most people do not
>>> build hbase, and the POM/jar/publishing thing is orthogonal - those
>>> who wish to build their own projects with ivy and/or ant are free to
>>> do so and not be impacted by our use of maven.
>>>
>>> We have ivy, but it doesnt integrate with our IDEs and is rather slow
>>> to build and rebuild.
>>>
>>> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
>>> <ma...@gmail.com> wrote:
>>>
>>>
>>>
>>>> -1
>>>>
>>>> I think Maven is too complex and will lower the adoption of HBase by
>>>> people today willing to build it.
>>>>
>>>> I would suggest using Ivy for dependency management as was done in
>>>> Thrift.
>>>>
>>>> Mathias.
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
Re: Moving to Maven (HBASE-2099)
Posted by Dhruba Borthakur <dh...@gmail.com>.
>From what I understand, the slowness of 'ivy' can be reduced if you can
fetch dependent jars from local ivy server, isn't it?
thanks,
dhruba
On Sat, Feb 13, 2010 at 12:25 AM, Kay Kay <ka...@gmail.com> wrote:
> Mathias -
> I have been using Ivy / Maven , interchangeably in different projects for
> the build management. Both of them clearly have their strong points and
> drawbacks. Ivy fits great for thrift because of the nature of tasks ,
> involved using some external command-line (thrift generators) etc. As I
> mentioned before - HBase does not have such cross maven goals / between the
> hairs as the build lifecycle is pretty straight-forward.
> In any case - the intention is to get to publish HBase artifacts and
> maintain a smaller core and encouraging contribs. from the artifacts as
> opposed to getting into the codebase.
> Once there are HBase artifacts published , the contrib / plugins for the
> same would be free to use ivy (with m2compatible="true") / maven as
> appropriate.
>
> Ryan -
> The slowness is attributed to the 'changing="true" ' in ivy.xml-s for all
> the hadoop-common / -hdfs / -mapreduce snapshots that we are using. I am
> facing similar 'slowness' with other mvn hadoop (snapshot) dependencies as
> well. In retrospective, that should have been made a configurable flag in
> libraries.properties , to ease things. Hopefully that is sorted out soon.
>
>
>
>
> On 02/13/2010 12:10 AM, Ryan Rawson wrote:
>
>> Would you mind elaborating more? At the moment, most people do not
>> build hbase, and the POM/jar/publishing thing is orthogonal - those
>> who wish to build their own projects with ivy and/or ant are free to
>> do so and not be impacted by our use of maven.
>>
>> We have ivy, but it doesnt integrate with our IDEs and is rather slow
>> to build and rebuild.
>>
>> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
>> <ma...@gmail.com> wrote:
>>
>>
>>> -1
>>>
>>> I think Maven is too complex and will lower the adoption of HBase by
>>> people today willing to build it.
>>>
>>> I would suggest using Ivy for dependency management as was done in
>>> Thrift.
>>>
>>> Mathias.
>>>
>>>
>>>
>>
>
--
Connect to me at http://www.facebook.com/dhruba
Re: Moving to Maven (HBASE-2099)
Posted by Kay Kay <ka...@gmail.com>.
Mathias -
I have been using Ivy / Maven , interchangeably in different
projects for the build management. Both of them clearly have their
strong points and drawbacks. Ivy fits great for thrift because of the
nature of tasks , involved using some external command-line (thrift
generators) etc. As I mentioned before - HBase does not have such cross
maven goals / between the hairs as the build lifecycle is pretty
straight-forward.
In any case - the intention is to get to publish HBase artifacts and
maintain a smaller core and encouraging contribs. from the artifacts as
opposed to getting into the codebase.
Once there are HBase artifacts published , the contrib / plugins for the
same would be free to use ivy (with m2compatible="true") / maven as
appropriate.
Ryan -
The slowness is attributed to the 'changing="true" ' in ivy.xml-s
for all the hadoop-common / -hdfs / -mapreduce snapshots that we are
using. I am facing similar 'slowness' with other mvn hadoop (snapshot)
dependencies as well. In retrospective, that should have been made a
configurable flag in libraries.properties , to ease things. Hopefully
that is sorted out soon.
On 02/13/2010 12:10 AM, Ryan Rawson wrote:
> Would you mind elaborating more? At the moment, most people do not
> build hbase, and the POM/jar/publishing thing is orthogonal - those
> who wish to build their own projects with ivy and/or ant are free to
> do so and not be impacted by our use of maven.
>
> We have ivy, but it doesnt integrate with our IDEs and is rather slow
> to build and rebuild.
>
> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
> <ma...@gmail.com> wrote:
>
>> -1
>>
>> I think Maven is too complex and will lower the adoption of HBase by
>> people today willing to build it.
>>
>> I would suggest using Ivy for dependency management as was done in Thrift.
>>
>> Mathias.
>>
>>
Re: Moving to Maven (HBASE-2099)
Posted by Ryan Rawson <ry...@gmail.com>.
Would you mind elaborating more? At the moment, most people do not
build hbase, and the POM/jar/publishing thing is orthogonal - those
who wish to build their own projects with ivy and/or ant are free to
do so and not be impacted by our use of maven.
We have ivy, but it doesnt integrate with our IDEs and is rather slow
to build and rebuild.
On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts
<ma...@gmail.com> wrote:
> -1
>
> I think Maven is too complex and will lower the adoption of HBase by
> people today willing to build it.
>
> I would suggest using Ivy for dependency management as was done in Thrift.
>
> Mathias.
>
Re: Moving to Maven (HBASE-2099)
Posted by Mathias Herberts <ma...@gmail.com>.
-1
I think Maven is too complex and will lower the adoption of HBase by
people today willing to build it.
I would suggest using Ivy for dependency management as was done in Thrift.
Mathias.
Re: Moving to Maven (HBASE-2099)
Posted by Ryan Rawson <ry...@gmail.com>.
Yes... a place to prep a patch and allow others to test.
On Feb 15, 2010 5:47 PM, "Kay Kay" <ka...@gmail.com> wrote:
Ryan -
So as I understand - this tree ( in github ) is some sort of workarea that
can afford to be broken / in a flux - during the transition (git mv /
build*.xml changes) we are talking about ?
On 2/15/10 4:33 PM, Paul Smith wrote:
>
> On 14/02/2010, at 3:42 PM, Ryan Rawson wrote:
>
>
>>...
Re: Moving to Maven (HBASE-2099)
Posted by Kay Kay <ka...@gmail.com>.
Ryan -
So as I understand - this tree ( in github ) is some sort of workarea
that can afford to be broken / in a flux - during the transition (git mv
/ build*.xml changes) we are talking about ?
On 2/15/10 4:33 PM, Paul Smith wrote:
> On 14/02/2010, at 3:42 PM, Ryan Rawson wrote:
>
>
>> here it is:
>>
>> http://github.com/ryanobjc/hbase/tree/trunk_maven
>>
>> let me know who wants access - github name.
>>
> Ryan, I've set myself up as 'tallpsmith' (my common moniker) on Github.
>
> I'm a git newbie really (have a good book though I'm reading now), so I apologise in advance if I get something wrong... this is actually a good excuse to get some git skillz.
>
> So, yeah, if you could grant me love in that tree, I would appreciate it, but watch out for newbie git mistakes... :)
>
> Paul
Re: Moving to Maven (HBASE-2099)
Posted by Paul Smith <ps...@aconex.com>.
On 14/02/2010, at 3:42 PM, Ryan Rawson wrote:
> here it is:
>
> http://github.com/ryanobjc/hbase/tree/trunk_maven
>
> let me know who wants access - github name.
Ryan, I've set myself up as 'tallpsmith' (my common moniker) on Github.
I'm a git newbie really (have a good book though I'm reading now), so I apologise in advance if I get something wrong... this is actually a good excuse to get some git skillz.
So, yeah, if you could grant me love in that tree, I would appreciate it, but watch out for newbie git mistakes... :)
Paul
Re: Moving to Maven (HBASE-2099)
Posted by Ryan Rawson <ry...@gmail.com>.
here it is:
http://github.com/ryanobjc/hbase/tree/trunk_maven
let me know who wants access - github name.
On Sat, Feb 13, 2010 at 6:49 PM, Ryan Rawson <ry...@gmail.com> wrote:
> I have previously uploaded most the hbase objects to github. I can give
> whomever commit access...
>
> On Feb 13, 2010 6:02 PM, "Paul Smith" <ps...@aconex.com> wrote:
>
>
> On 13/02/2010, at 6:59 PM, Kay Kay wrote:
>
>> +1 on the 'maven'ization.
>>
>> As far as I see - there...
>
> Yep, I think HBase definitely is a standard project type that fits the
> intent that Maven was originally designed for. There's no question, if a
> project has a non-standard requirement, then going down a Maven path could
> be tricky. But HBase is nice and simple, and standard.
>
>> As I had suggested earlier - the best course of action, for a smooth
>> transition would be -
>>
>> *...
>
> sounds good, so keep the ivy part tracking the Maven changes (basically a
> side-by-side approach) until a final switch?
>
> It might be worth setting up a branch too for visibility prior to a
> commit/merge to trunk (I don't recommend a merge under SVN for sure, GIT
> probably handles that way better).
>
> Paul
Re: Moving to Maven (HBASE-2099)
Posted by Ryan Rawson <ry...@gmail.com>.
I have previously uploaded most the hbase objects to github. I can give
whomever commit access...
On Feb 13, 2010 6:02 PM, "Paul Smith" <ps...@aconex.com> wrote:
On 13/02/2010, at 6:59 PM, Kay Kay wrote:
> +1 on the 'maven'ization.
>
> As far as I see - there...
Yep, I think HBase definitely is a standard project type that fits the
intent that Maven was originally designed for. There's no question, if a
project has a non-standard requirement, then going down a Maven path could
be tricky. But HBase is nice and simple, and standard.
> As I had suggested earlier - the best course of action, for a smooth
transition would be -
>
> *...
sounds good, so keep the ivy part tracking the Maven changes (basically a
side-by-side approach) until a final switch?
It might be worth setting up a branch too for visibility prior to a
commit/merge to trunk (I don't recommend a merge under SVN for sure, GIT
probably handles that way better).
Paul
Re: Moving to Maven (HBASE-2099)
Posted by Paul Smith <ps...@aconex.com>.
On 13/02/2010, at 6:59 PM, Kay Kay wrote:
> +1 on the 'maven'ization.
>
> As far as I see - there is not any special cross goal dependencies coming up, in the HBase lifecycle so this should not be a problem.
>
Yep, I think HBase definitely is a standard project type that fits the intent that Maven was originally designed for. There's no question, if a project has a non-standard requirement, then going down a Maven path could be tricky. But HBase is nice and simple, and standard.
> As I had suggested earlier - the best course of action, for a smooth transition would be -
>
> * directory changes as necessary for mvn
> * modify build scripts.
> .
> .
> .
> * working pom.xml in place and build scripts /ivy removed.
sounds good, so keep the ivy part tracking the Maven changes (basically a side-by-side approach) until a final switch?
It might be worth setting up a branch too for visibility prior to a commit/merge to trunk (I don't recommend a merge under SVN for sure, GIT probably handles that way better).
Paul
Re: Moving to Maven (HBASE-2099)
Posted by Kay Kay <ka...@gmail.com>.
+1 on the 'maven'ization.
As far as I see - there is not any special cross goal dependencies
coming up, in the HBase lifecycle so this should not be a problem.
As I had suggested earlier - the best course of action, for a smooth
transition would be -
* directory changes as necessary for mvn
* modify build scripts.
.
.
.
* working pom.xml in place and build scripts /ivy removed.
Good work, Paul .
On 02/12/2010 06:10 PM, Paul Smith wrote:
> I while ago I prepared a fairly lengthy email to this list about the potential of moving HBase to a Maven build system (as is discussed on HBASE-2099). On reflection, it was a) long, and b) a bit over the top. I think from a small smattering of discussions on IRC it appears there's at least a view that it might be worth kicking off a discussion, so the content of the original mail isn't needed.
>
> So, lets talk! :)
>
> I personally think Maven will assist as the HBase community to refactor the Hbase system into various components and should make your lives a lot easier. The built-in reporting and packaging elements make things easy if you follow conventions (I'm just proposing a few directory shuffles to follow these conventions to simplify things). I'm also recommending a slight change in direction with respect to the produced tar balls that are spit out.
>
> What do people think? The IntelliJ built in support for Maven is outstanding, and the M2Eclipse plugin is excellent for Eclipse, so there's good support for it in the IDE. Kay Kay looks to have good Maven skills up his sleeve by the looks of his work on HBASE-2212 moving out the Lucene code to a github project (nice work!), so there should be good support around. Maven will help simplify the publishing of artifacts to the Maven repositories (particularly regular snapshots for anyone upstream that want's easy access to the latest without having to build from source).
>
> HBASE-2099 has a lengthy comment trail on it, so may not be easily digestible. If you have any questions or concerns I'd really like to hear your thoughts. If you're unfamiliar with Maven I highly recommend the Maven: The Definitive Guide (free, see [1])
>
> cheers,
>
> Paul
>
> [1] Maven: The Definitive Guide - http://www.sonatype.com/book/
> (now split into 2 books, a by-example, and a reference guide).
>
>
>
Re: Moving to Maven (HBASE-2099)
Posted by Stack <st...@duboce.net>.
I'm +1 (I've used maven in past lives).
Paul: Any chance of publishing the website up on your personal apache
account and passing out the link to show folks what a mavenized hbase
website would look like. In particular the reports you mention in the
below.
Good stuff,
St.Ack
On Fri, Feb 12, 2010 at 6:10 PM, Paul Smith <ps...@aconex.com> wrote:
> I while ago I prepared a fairly lengthy email to this list about the potential of moving HBase to a Maven build system (as is discussed on HBASE-2099). On reflection, it was a) long, and b) a bit over the top. I think from a small smattering of discussions on IRC it appears there's at least a view that it might be worth kicking off a discussion, so the content of the original mail isn't needed.
>
> So, lets talk! :)
>
> I personally think Maven will assist as the HBase community to refactor the Hbase system into various components and should make your lives a lot easier. The built-in reporting and packaging elements make things easy if you follow conventions (I'm just proposing a few directory shuffles to follow these conventions to simplify things). I'm also recommending a slight change in direction with respect to the produced tar balls that are spit out.
>
> What do people think? The IntelliJ built in support for Maven is outstanding, and the M2Eclipse plugin is excellent for Eclipse, so there's good support for it in the IDE. Kay Kay looks to have good Maven skills up his sleeve by the looks of his work on HBASE-2212 moving out the Lucene code to a github project (nice work!), so there should be good support around. Maven will help simplify the publishing of artifacts to the Maven repositories (particularly regular snapshots for anyone upstream that want's easy access to the latest without having to build from source).
>
> HBASE-2099 has a lengthy comment trail on it, so may not be easily digestible. If you have any questions or concerns I'd really like to hear your thoughts. If you're unfamiliar with Maven I highly recommend the Maven: The Definitive Guide (free, see [1])
>
> cheers,
>
> Paul
>
> [1] Maven: The Definitive Guide - http://www.sonatype.com/book/
> (now split into 2 books, a by-example, and a reference guide).
>
>
>
Re: Moving to Maven (HBASE-2099)
Posted by Dan Washusen <da...@reactive.org>.
+1
Maven + IntelliJ is awesome! I currently use a shell script to publish
snapshots of HBase to our repository. It works but it's a royal pain in
backside to maintain... :)
I'd also love a small client jar that contains as few classes as possible.
Especially not the jetty stuff! We use embedded jetty and currently have to
modify the hbase jar to remove refences to jetty so our app starts...
On 13 February 2010 13:27, Ryan Rawson <ry...@gmail.com> wrote:
> Do it. I love intellij and would totally test. Fast builds ftw.
>
> On Feb 12, 2010 6:11 PM, "Paul Smith" <ps...@aconex.com> wrote:
>
> I while ago I prepared a fairly lengthy email to this list about the
> potential of moving HBase to a Maven build system (as is discussed on
> HBASE-2099). On reflection, it was a) long, and b) a bit over the top. I
> think from a small smattering of discussions on IRC it appears there's at
> least a view that it might be worth kicking off a discussion, so the
> content of the original mail isn't needed.
>
> So, lets talk! :)
>
> I personally think Maven will assist as the HBase community to refactor the
> Hbase system into various components and should make your lives a lot
> easier. The built-in reporting and packaging elements make things easy if
> you follow conventions (I'm just proposing a few directory shuffles to
> follow these conventions to simplify things). I'm also recommending a
> slight change in direction with respect to the produced tar balls that are
> spit out.
>
> What do people think? The IntelliJ built in support for Maven is
> outstanding, and the M2Eclipse plugin is excellent for Eclipse, so there's
> good support for it in the IDE. Kay Kay looks to have good Maven skills
> up his sleeve by the looks of his work on HBASE-2212 moving out the Lucene
> code to a github project (nice work!), so there should be good support
> around. Maven will help simplify the publishing of artifacts to the Maven
> repositories (particularly regular snapshots for anyone upstream that
> want's
> easy access to the latest without having to build from source).
>
> HBASE-2099 has a lengthy comment trail on it, so may not be easily
> digestible. If you have any questions or concerns I'd really like to hear
> your thoughts. If you're unfamiliar with Maven I highly recommend the
> Maven: The Definitive Guide (free, see [1])
>
> cheers,
>
> Paul
>
> [1] Maven: The Definitive Guide - http://www.sonatype.com/book/
> (now split into 2 books, a by-example, and a reference guide).
>
Re: Moving to Maven (HBASE-2099)
Posted by Ryan Rawson <ry...@gmail.com>.
Do it. I love intellij and would totally test. Fast builds ftw.
On Feb 12, 2010 6:11 PM, "Paul Smith" <ps...@aconex.com> wrote:
I while ago I prepared a fairly lengthy email to this list about the
potential of moving HBase to a Maven build system (as is discussed on
HBASE-2099). On reflection, it was a) long, and b) a bit over the top. I
think from a small smattering of discussions on IRC it appears there's at
least a view that it might be worth kicking off a discussion, so the
content of the original mail isn't needed.
So, lets talk! :)
I personally think Maven will assist as the HBase community to refactor the
Hbase system into various components and should make your lives a lot
easier. The built-in reporting and packaging elements make things easy if
you follow conventions (I'm just proposing a few directory shuffles to
follow these conventions to simplify things). I'm also recommending a
slight change in direction with respect to the produced tar balls that are
spit out.
What do people think? The IntelliJ built in support for Maven is
outstanding, and the M2Eclipse plugin is excellent for Eclipse, so there's
good support for it in the IDE. Kay Kay looks to have good Maven skills
up his sleeve by the looks of his work on HBASE-2212 moving out the Lucene
code to a github project (nice work!), so there should be good support
around. Maven will help simplify the publishing of artifacts to the Maven
repositories (particularly regular snapshots for anyone upstream that want's
easy access to the latest without having to build from source).
HBASE-2099 has a lengthy comment trail on it, so may not be easily
digestible. If you have any questions or concerns I'd really like to hear
your thoughts. If you're unfamiliar with Maven I highly recommend the
Maven: The Definitive Guide (free, see [1])
cheers,
Paul
[1] Maven: The Definitive Guide - http://www.sonatype.com/book/
(now split into 2 books, a by-example, and a reference guide).