hbase-specific maven dependencies

[Jesse Yates <je...@gmail.com>]

Hey all,

I was poking around the pom yesterday and noticed that we are hosting our
own special artifacts (e.g. our custom surefire plugin via Keywal's)  in
various commiters' apache 'mvn' repositories. For instance, the surefire
plugin is hosted here:
*
    <repository>
      <id>ghelmling.testing</id>
      <name>Gary Helmling test repo</name>
      <url>http://people.apache.org/~garyh/mvn/</url>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <releases>
        <enabled>true</enabled>
      </releases>
    </repository>
*

(And so I'm not just picking on Gary, Stack has one too :) for hosting
asynchbase, which is another discussion as to whether or not we should be
doing *that*).

This leads to a lack of visibility and flexiblity into the objects being
deployed - we are dependent on the committer to put up the right version
and their benevolence when we want to change the artifact, all with without
any history of what they were doing (not saying we don't trust you fellas).

I'd like to propose that we start storing these built artifacts either:

   1. In a branch of hbase that is web-accessible and just reference that
   branch as a maven repo
      1. I'm (perhaps foolishly) volunteering to setup the pom to make this
      happen
   2. In one of the free sonatype respositories that we can name
   hbase-custom-dependencies (or some such).

The former has the advantage that we have full control over the repository
as well as commit history on that branch to audit changes.
The latter is nice in that it's an official repo (they are supposedly free
for OSS projects), but we loose a bit of the visibility into changes and
its a lot more friction to setup.

Personally I'm predisposed to (1), especially because they are just
hbase-specific dependencies. Any projects depending on hbase via maven will
automatically pull in the right dependencies, without the micro-repository
pain<http://www.thewebsemantic.com/2009/04/11/your-very-own-google-code-maven-repo/comment-page-1/#comment-1312>
.

Thoughts?

Thanks,
Jesse

-------------------
Jesse Yates
@jesse_yates
jyates.github.com

Re: hbase-specific maven dependencies

[Stack <st...@duboce.net>]

On Thu, Sep 27, 2012 at 1:41 PM, Jesse Yates <je...@gmail.com> wrote:
> On Thu, Sep 27, 2012 at 12:24 PM, Gary Helmling <gh...@gmail.com> wrote:
>
>> See HBASE-4955.  This was never seen as a permanent feature, but
>> rather as a stop gap measure (though it's now been around for a
>> while).  Nicolas has been working hard to get these changes in
>> upstream, but we're still blocked waiting on official releases.  Once
>> those are out, this repo is no longer necessary.
>>
>> There are 2 issues here:
>> * the hosting of the build artifacts as dependencies
>> * the hosting of sources used to build those artifacts
>>
>>
> My concern is that while in this case it was a temporary thing, its
> something that we do with some frequency. And the fact that we have two
> different repositories already (yours and stacks). It would be better to
> consolidate and put it under source control.
>

I just removed mine.  Was a leftover from another tmp hosting while we
waited on a release.
St.Ack

Re: hbase-specific maven dependencies

[n keywal <nk...@gmail.com>]

I totally agree with Gary.

We can industrialize the way we manage internally a custom dependency, but
the most important part is being able to come back to the official release,
and this part can't be industrialized as it does not depend on us...

Nicolas

On Fri, Sep 28, 2012 at 8:11 PM, Gary Helmling <gh...@gmail.com> wrote:

> > My concern is that while in this case it was a temporary thing, its
> > something that we do with some frequency. And the fact that we have two
> > different repositories already (yours and stacks). It would be better to
> > consolidate and put it under source control.
> >
>
> I'm sure others would say the fact that we do this with some frequency
> is the problem, that the custom built dependencies are the real issue.
>  And moving them from one repo to another doesn't solve that.
>
> If this is going to be an ongoing need for the future, with new custom
> built dependencies coming up then I can see the benefit of a shared
> resource.  But personally I hope that we can actually remove the need
> for my repo much sooner than later.  And forking our own patched
> builds doesn't really fit so well with the Apache way, which I think
> would prefer that we work the the other communities to get the changes
> we need released.  (Yes, I'm arguing this even though I'm hosting our
> current patched builds, because I think it's best if we avoid going
> this route in the future.)
>
> >>
> >> If someone else wants to setup a new repo for hosting these they are
> >> welcome to, but personally I don't see much of a benefit.
> >
> >
> > A single unified location and ability to see easily version changes,
> > updates, etc would be a start. Cleaning up the pom is another.
> >
> >
>
> Until we have new needs for custom builds, which we should look at on
> a case-by-case basis anyway, I don't really see this applying.  Until
> then it seems like duplicating work that's already gone in to
> producing and deploying the existing builds.
>
> >>
> >> If the problem is just accessibility to other committers, another
> >> hacky solution is that I just make my ~garyh/mvn directory writable by
> >> the hbase group so that other committers could make changes if
> >> necessary.
> >>
> >
> > That's a start, but I don't think solves all our problems. Also, I heard
> > from LarsH that apache was planning on taking away those directories?
> >
>
> As I understand it, these were entirely separate committer repos being
> removed, unrelated to our (ab)use of public HTML access on
> people.apache.org.
>
>
> If you see enough benefit here to invest the effort then you are of
> course welcome to do so.  I'd be happy to see use of my repo removed.
> But if all we're doing is moving our existing custom builds from one
> place to another, then that just seems like a duplication of effort
> that's already been sunk, without getting us closer to the end outcome
> of eliminating the custom builds and using actual releases.  But
> everyone on this project is free to invest effort where they see fit.
>
> --gh
>

Re: hbase-specific maven dependencies

[Gary Helmling <gh...@gmail.com>]

> My concern is that while in this case it was a temporary thing, its
> something that we do with some frequency. And the fact that we have two
> different repositories already (yours and stacks). It would be better to
> consolidate and put it under source control.
>

I'm sure others would say the fact that we do this with some frequency
is the problem, that the custom built dependencies are the real issue.
 And moving them from one repo to another doesn't solve that.

If this is going to be an ongoing need for the future, with new custom
built dependencies coming up then I can see the benefit of a shared
resource.  But personally I hope that we can actually remove the need
for my repo much sooner than later.  And forking our own patched
builds doesn't really fit so well with the Apache way, which I think
would prefer that we work the the other communities to get the changes
we need released.  (Yes, I'm arguing this even though I'm hosting our
current patched builds, because I think it's best if we avoid going
this route in the future.)

>>
>> If someone else wants to setup a new repo for hosting these they are
>> welcome to, but personally I don't see much of a benefit.
>
>
> A single unified location and ability to see easily version changes,
> updates, etc would be a start. Cleaning up the pom is another.
>
>

Until we have new needs for custom builds, which we should look at on
a case-by-case basis anyway, I don't really see this applying.  Until
then it seems like duplicating work that's already gone in to
producing and deploying the existing builds.

>>
>> If the problem is just accessibility to other committers, another
>> hacky solution is that I just make my ~garyh/mvn directory writable by
>> the hbase group so that other committers could make changes if
>> necessary.
>>
>
> That's a start, but I don't think solves all our problems. Also, I heard
> from LarsH that apache was planning on taking away those directories?
>

As I understand it, these were entirely separate committer repos being
removed, unrelated to our (ab)use of public HTML access on
people.apache.org.


If you see enough benefit here to invest the effort then you are of
course welcome to do so.  I'd be happy to see use of my repo removed.
But if all we're doing is moving our existing custom builds from one
place to another, then that just seems like a duplication of effort
that's already been sunk, without getting us closer to the end outcome
of eliminating the custom builds and using actual releases.  But
everyone on this project is free to invest effort where they see fit.

--gh

Re: hbase-specific maven dependencies

[Jesse Yates <je...@gmail.com>]

On Thu, Sep 27, 2012 at 12:24 PM, Gary Helmling <gh...@gmail.com> wrote:

> See HBASE-4955.  This was never seen as a permanent feature, but
> rather as a stop gap measure (though it's now been around for a
> while).  Nicolas has been working hard to get these changes in
> upstream, but we're still blocked waiting on official releases.  Once
> those are out, this repo is no longer necessary.
>
> There are 2 issues here:
> * the hosting of the build artifacts as dependencies
> * the hosting of sources used to build those artifacts
>
>
My concern is that while in this case it was a temporary thing, its
something that we do with some frequency. And the fact that we have two
different repositories already (yours and stacks). It would be better to
consolidate and put it under source control.



> The source code for the artifacts is available at:
>
> Junit: 4.10-HBASE-1, source code on https://github.com/nkeywal/junit,
> branch "hbase"
> Surefire: 2.12-TRUNK-HBASE-2, source code on
> https://github.com/nkeywal/maven-surefire
>
>
I'm fine with the source being somewhere else, as long as the pom for that
artifact has that information in it, so interested parties can know where
to look.

When we do host special artifacts, we should also put up a -source tarball
too so IDEs can pull down those sources automatically.


>
> >
> >    1. In a branch of hbase that is web-accessible and just reference that
> >    branch as a maven repo
> >       1. I'm (perhaps foolishly) volunteering to setup the pom to make
> this
> >       happen
>
> So making the ASF svn repo the maven repo instead?  That seems like a
> weird hack to me, though I suppose it would at least be accessible to
> others.
>

All a maven repo is is just a remotely accessible directory with a certain
structure - nothing fancy.


>
> >    2. In one of the free sonatype respositories that we can name
> >    hbase-custom-dependencies (or some such).
> >
>
> Who would own this and how would we control access?


I'd assume its globally owned by comitters and they have the credentials.
But then we already have that with svn....


>  Is there any ASF
> resource that we can use here instead?


No idea.


>
> If someone else wants to setup a new repo for hosting these they are
> welcome to, but personally I don't see much of a benefit.


A single unified location and ability to see easily version changes,
updates, etc would be a start. Cleaning up the pom is another.


> Are there
> build changes we'd like to make that this is blocking?
>

Not that I know of. It just seemed like a nice cleanup.


>
> If the problem is just accessibility to other committers, another
> hacky solution is that I just make my ~garyh/mvn directory writable by
> the hbase group so that other committers could make changes if
> necessary.
>

That's a start, but I don't think solves all our problems. Also, I heard
from LarsH that apache was planning on taking away those directories?


>
> But I think the best solution is to get the upstream releases out so
> that we can eliminate the custom dependency repos entirely.
> Unfortunately we don't have much control over that.
>

Agreed.

Re: hbase-specific maven dependencies

[Gary Helmling <gh...@gmail.com>]

> I was poking around the pom yesterday and noticed that we are hosting our
> own special artifacts (e.g. our custom surefire plugin via Keywal's)  in
> various commiters' apache 'mvn' repositories. For instance, the surefire
> plugin is hosted here:
> *
>     <repository>
>       <id>ghelmling.testing</id>
>       <name>Gary Helmling test repo</name>
>       <url>http://people.apache.org/~garyh/mvn/</url>
>       <snapshots>
>         <enabled>true</enabled>
>       </snapshots>
>       <releases>
>         <enabled>true</enabled>
>       </releases>
>     </repository>
> *
>
> (And so I'm not just picking on Gary, Stack has one too :) for hosting
> asynchbase, which is another discussion as to whether or not we should be
> doing *that*).
>

See HBASE-4955.  This was never seen as a permanent feature, but
rather as a stop gap measure (though it's now been around for a
while).  Nicolas has been working hard to get these changes in
upstream, but we're still blocked waiting on official releases.  Once
those are out, this repo is no longer necessary.

There are 2 issues here:
* the hosting of the build artifacts as dependencies
* the hosting of sources used to build those artifacts

The source code for the artifacts is available at:

Junit: 4.10-HBASE-1, source code on https://github.com/nkeywal/junit,
branch "hbase"
Surefire: 2.12-TRUNK-HBASE-2, source code on
https://github.com/nkeywal/maven-surefire


>
>    1. In a branch of hbase that is web-accessible and just reference that
>    branch as a maven repo
>       1. I'm (perhaps foolishly) volunteering to setup the pom to make this
>       happen

So making the ASF svn repo the maven repo instead?  That seems like a
weird hack to me, though I suppose it would at least be accessible to
others.

>    2. In one of the free sonatype respositories that we can name
>    hbase-custom-dependencies (or some such).
>

Who would own this and how would we control access?  Is there any ASF
resource that we can use here instead?

If someone else wants to setup a new repo for hosting these they are
welcome to, but personally I don't see much of a benefit.  Are there
build changes we'd like to make that this is blocking?

If the problem is just accessibility to other committers, another
hacky solution is that I just make my ~garyh/mvn directory writable by
the hbase group so that other committers could make changes if
necessary.

But I think the best solution is to get the upstream releases out so
that we can eliminate the custom dependency repos entirely.
Unfortunately we don't have much control over that.