You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Drew Farris <dr...@gmail.com> on 2010/01/19 16:48:40 UTC

Hadoop snapshot dependency & source jars.

Zhao is looking for the sources for the current hadoop dependency,
which is currently set to 0.20.2-SNAPSHOT. The hadoop-core project
doesn't currently publish source attachments alongside their
snapshots, so the only option would be to pull sources from the
hadoop-core svn repo using the datestamp of the latest snapshot. This
is a pain to maintain because in mahout the hadoop-core snapshots get
updated on an automatic basis.

I've been digging through the hadoop mailing list archives and can't
find any discussion of publishing attached sources. Is anyone in this
group that familiar enough with this aspect of the hadoop dev process
to comment? If not, I'll go ahead and ask about it on the hadoop-dev
list.

Mahout should also depend on a fixed, point in time snapshot which
would make the maintenance of a personal copy of the sources easier.
This would cause less problems for mahout devs because the project
would not be building against a moving target and new issues that
potentially appear.

It is my understanding that we can't depend on 0.20.1 due do bugs in
the release version that have been fixed in the snapshots.

Drew

Re: Hadoop snapshot dependency & source jars.

Posted by Drew Farris <dr...@gmail.com>.
On Tue, Jan 19, 2010 at 11:57 AM, Olivier Grisel
<ol...@ensta.org> wrote:
>
> AFAIK, most maven repository servers do not keep the complete history
> of SNAPSHOT versions, just the 3 latest successfully deployed builds
> or so.
>

Ahh, that's right. So a static timestamp won't work for us at all
because the dependency will disappear from the repo.

Re: Hadoop snapshot dependency & source jars.

Posted by Olivier Grisel <ol...@ensta.org>.
2010/1/19 Drew Farris <dr...@gmail.com>:
> On Tue, Jan 19, 2010 at 10:55 AM, Benson Margulies
> <bi...@gmail.com> wrote:
>> So, ideally, you'd convince hadoop to turn on timestamped snapshots
>> and to publish source.
>
> Published sources yes, timestamped snapshots are already available, it
> is just a matter of switching the mahout parent pom from
> 0.20.2-SNAPSHOT to 0.20.2-20100112.125701-4 or whatever.

AFAIK, most maven repository servers do not keep the complete history
of SNAPSHOT versions, just the 3 latest successfully deployed builds
or so.

-- 
Olivier
http://twitter.com/ogrisel - http://code.oliviergrisel.name

Re: Hadoop snapshot dependency & source jars.

Posted by Drew Farris <dr...@gmail.com>.
On Tue, Jan 19, 2010 at 10:55 AM, Benson Margulies
<bi...@gmail.com> wrote:
> So, ideally, you'd convince hadoop to turn on timestamped snapshots
> and to publish source.

Published sources yes, timestamped snapshots are already available, it
is just a matter of switching the mahout parent pom from
0.20.2-SNAPSHOT to 0.20.2-20100112.125701-4 or whatever.

> Failing that, and somehow I always end up in the role of proposing
> this sort of chewing gum and bailing wire, we could make a branch of
> them, set the versions to be whatever-mahout, and publish our own
> snapshots....

Right, and this is how it used to work prior to the 0.2-release,
although if it does come to this, this time around we should make sure
the pom we deploy has the dependencies included.

Re: Hadoop snapshot dependency & source jars.

Posted by Benson Margulies <bi...@gmail.com>.
So, ideally, you'd convince hadoop to turn on timestamped snapshots
and to publish source.

Failing that, and somehow I always end up in the role of proposing
this sort of chewing gum and bailing wire, we could make a branch of
them, set the versions to be whatever-mahout, and publish our own
snapshots....

On Tue, Jan 19, 2010 at 10:48 AM, Drew Farris <dr...@gmail.com> wrote:
> Zhao is looking for the sources for the current hadoop dependency,
> which is currently set to 0.20.2-SNAPSHOT. The hadoop-core project
> doesn't currently publish source attachments alongside their
> snapshots, so the only option would be to pull sources from the
> hadoop-core svn repo using the datestamp of the latest snapshot. This
> is a pain to maintain because in mahout the hadoop-core snapshots get
> updated on an automatic basis.
>
> I've been digging through the hadoop mailing list archives and can't
> find any discussion of publishing attached sources. Is anyone in this
> group that familiar enough with this aspect of the hadoop dev process
> to comment? If not, I'll go ahead and ask about it on the hadoop-dev
> list.
>
> Mahout should also depend on a fixed, point in time snapshot which
> would make the maintenance of a personal copy of the sources easier.
> This would cause less problems for mahout devs because the project
> would not be building against a moving target and new issues that
> potentially appear.
>
> It is my understanding that we can't depend on 0.20.1 due do bugs in
> the release version that have been fixed in the snapshots.
>
> Drew
>

Re: Hadoop snapshot dependency & source jars.

Posted by Drew Farris <dr...@gmail.com>.
I found the >right< archives and the issue was brought up as recently
as 1/4 with no responses. I'll see what I can do for a patch.

On Tue, Jan 19, 2010 at 1:17 PM, Drew Farris <dr...@gmail.com> wrote:
> On Tue, Jan 19, 2010 at 12:56 PM, Grant Ingersoll <gs...@apache.org> wrote:
>> Should be easy enough to submit a patch to Hadoop, no?  I can't imagine they are against it, probably just that no one has done it.
>
> I'll post to the hadoop-dev list and see what I can discover.
>

Re: Hadoop snapshot dependency & source jars.

Posted by Drew Farris <dr...@gmail.com>.
On Tue, Jan 19, 2010 at 12:56 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Should be easy enough to submit a patch to Hadoop, no?  I can't imagine they are against it, probably just that no one has done it.

I'll post to the hadoop-dev list and see what I can discover.

Re: Hadoop snapshot dependency & source jars.

Posted by Jake Mannix <ja...@gmail.com>.
On Tue, Jan 19, 2010 at 10:02 AM, Benson Margulies <bi...@gmail.com>wrote:

> Well, there are two different problems floating around here. Problem 1
> is essentially that hadoop doesn't release often enough.
>

We're really not in much of a position to throw that stone in this glass
house... :P

  -jake

Re: Hadoop snapshot dependency & source jars.

Posted by Benson Margulies <bi...@gmail.com>.
Well, there are two different problems floating around here. Problem 1
is essentially that hadoop doesn't release often enough. Problem 2 is
the lack of source on the snapshots.

If you fix #2 with a patch, #1 will still byte you.


On Tue, Jan 19, 2010 at 12:56 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Should be easy enough to submit a patch to Hadoop, no?  I can't imagine they are against it, probably just that no one has done it.
>
> On Jan 19, 2010, at 10:48 AM, Drew Farris wrote:
>
>> Zhao is looking for the sources for the current hadoop dependency,
>> which is currently set to 0.20.2-SNAPSHOT. The hadoop-core project
>> doesn't currently publish source attachments alongside their
>> snapshots, so the only option would be to pull sources from the
>> hadoop-core svn repo using the datestamp of the latest snapshot. This
>> is a pain to maintain because in mahout the hadoop-core snapshots get
>> updated on an automatic basis.
>>
>> I've been digging through the hadoop mailing list archives and can't
>> find any discussion of publishing attached sources. Is anyone in this
>> group that familiar enough with this aspect of the hadoop dev process
>> to comment? If not, I'll go ahead and ask about it on the hadoop-dev
>> list.
>>
>> Mahout should also depend on a fixed, point in time snapshot which
>> would make the maintenance of a personal copy of the sources easier.
>> This would cause less problems for mahout devs because the project
>> would not be building against a moving target and new issues that
>> potentially appear.
>>
>> It is my understanding that we can't depend on 0.20.1 due do bugs in
>> the release version that have been fixed in the snapshots.
>>
>> Drew
>
>
>

Re: Hadoop snapshot dependency & source jars.

Posted by Grant Ingersoll <gs...@apache.org>.
Should be easy enough to submit a patch to Hadoop, no?  I can't imagine they are against it, probably just that no one has done it.

On Jan 19, 2010, at 10:48 AM, Drew Farris wrote:

> Zhao is looking for the sources for the current hadoop dependency,
> which is currently set to 0.20.2-SNAPSHOT. The hadoop-core project
> doesn't currently publish source attachments alongside their
> snapshots, so the only option would be to pull sources from the
> hadoop-core svn repo using the datestamp of the latest snapshot. This
> is a pain to maintain because in mahout the hadoop-core snapshots get
> updated on an automatic basis.
> 
> I've been digging through the hadoop mailing list archives and can't
> find any discussion of publishing attached sources. Is anyone in this
> group that familiar enough with this aspect of the hadoop dev process
> to comment? If not, I'll go ahead and ask about it on the hadoop-dev
> list.
> 
> Mahout should also depend on a fixed, point in time snapshot which
> would make the maintenance of a personal copy of the sources easier.
> This would cause less problems for mahout devs because the project
> would not be building against a moving target and new issues that
> potentially appear.
> 
> It is my understanding that we can't depend on 0.20.1 due do bugs in
> the release version that have been fixed in the snapshots.
> 
> Drew