ASF Repository Release issues (was Re: Repository for snapshots)

["Mark R. Diggory" <md...@latte.harvard.edu>]

I'm going to cc this discussion onto infrastructure. I think it is a
valid topic which we need to address there in more detail.

Infrastructure, please review this posting before responding to this
subject:

http://article.gmane.org/gmane.comp.jakarta.commons.devel/39469


Jason van Zyl wrote:
> On Sat, 2004-07-17 at 21:03, Brett Porter wrote:
> 
>> I don't know very much about cvs.apache.org/repository.
>> 
>> What I was suggesting was that snapshots that we want to keep 
>> should go to /dist/java-repository (which is what happens now).
> 
> 
> Cool, that's all that's needed then. Whatever a projects wishes to 
> store on Ibiblio should be stored on Ibiblio. That's all I want to 
> happen. If that can happen with the current setup then that's good.
> 
> 

I'm not going to suggest the current setup is optimal, its not going to
meet everyones needs. The infrastructure issues are valid. And you have
to consider what the nature of the "www.apache.org/dist directory is in
terms of the ASF Repository being there.

My goal in placing it into dist was to actually supersede dist in the
long run with the ASF Repository Structure defined in the Specification.

http://wiki.apache.org/old/ASFRepository
http://nagoya.apache.org/wiki/apachewiki.cgi?ASFRepository/URISyntax

This is something some may consider aggressive. Eventually we should
actually have a ASF Repository structure that all Apache releases go
into (even if it is simply a project placing their stuff into
www.apache.org/dist/project/...

Most projects out there do maintain major version releases in dist, and
if they are removed from the projects releases directory, its upto the
mirror to decide its cleanup strategy in rsync, this means some mirrors
will not delete copies, while others will. I beleive archive.apache.org
is meant to be such a mirror that maintains all the content that was
ever in the www.apache.org/dist directory.

As such, IMHO, the restictions placed on dist are to reduce the
archiving of any unofficial releases. So, IMHO placing any "dated build"
releases into the ASF Repository violates the requirements because they
will be "archived" and they are not a full release.

I recognize its almost impossible for Infrustructure to enforce this
policy; for example; managing the release of every single maven plugin
within Infrustructure is probibly not in their interest. What is in
their interest is being able to maintain a secure release directory with
properly signed releases. I'm convinced we are at a point where
Infrastructure needs to reflect on the current release strategy and
attempt to open it up so that we can manage more complex release
scenarios. We also need to establish a course of action to migrate the
directory structures to that structure defined by the ASF Repository Spec.

The battle here for us is to maintain "group cohesion" between all the
Apache projects in terms of where and how to release content, no matter
how painful interaction can become. So, pushing content off to a
different location isn't very beneficial in this regard. It may be
tedious and frustrating to deal with, but we need to maintain such
interaction.

What this means to me is that we have a few possible solutions for the
future direction of ASF Repository:

1.) do not allow any interim content into www/dist. Interm content gets
placed under cvs.apache.org/repository. This gives us a means to allow
testing of interim content while not having all the mirrors (including
ibibilio and the archive) get flooded with it.

2.) Establish a means to filter what gets into the archive (and possibly
the mirrors). Store all content under one repository location (in dist).
This is allot more complex and will require effort across multiple
projects and infrastructure to establish as a practice. (I'm not
convinced this will be possible).

3.) Establish the ASF Repository independent of the www/dist directory
and manage all the content within it, providing our own mechanisms for
archiving and allowing interim builds. Initial structure will be "Maven"
like, migrating however, to the ASF Repository Specification. The
problem is that this fractures the community into those that want to
release in the old directory structure and those that want to release in
the Repository.

So, theres allot of discussion and voting that needs to continue on this
subject to determine the best course of action, I think that the current
compromise is not an effective solution for the long term, It reflects my
initial efforts to establish a repository for ASF projects and now that
we have our footing we need more member involvement and decision making
to establish the proper direction for this to go into. This kind of
growth is hard, changing the way a project releases is a project in and
of itself.

-Mark Diggory


>> Quoting Jason van Zyl <jv...@maven.org>:
>> 
>>> On Sat, 2004-07-17 at 19:46, Brett Porter wrote:
>>> 
>>>> Nightly builds go to http://cvs.apache.org/repository, which is
>>>>  something we don't do at the moment, and something I don't 
>>>> think needs to be synced out to the mirrors. My understanding 
>>>> is that the ASF for legal reasons needs a PMC vote on a release
>>>>  to have it officially released, and automated builds are 
>>>> obviously not that.
>>>> 
>>>> I think what we have at the moment is fine.
>>> 
>>> If someone wishes to have a snapshot persist does this happen? If
>>>  the snapshots are not archived in perpetuity then it's not fine.
>>>  A project should be able to have last what it desires to last. 
>>> If the Apache repository where snapshots go lives on that's cool,
>>>  otherwise something else needs to be done. Either the PMC for a
>>>  project should just be able to say snapshots between versions
>>> are fine for release to ibiblio or the snapshots place in the
>>> Apache repo for this must persist. I agree that storing nightlies
>>> might be a little excessive but if a project makes a snapshot per
>>> week and wishes it to live on at Ibiblio they should be able to
>>> do this.
>>> 
>>> Do you know what the lifespan is for snapshots stored in 
>>> cvs.apache.org/repository
>>> 
>>> ?
>>> 
>>> 
>>>> Cheers, Brett
>>>> 
>>>> Quoting Carlos Sanchez <ap...@carlos.cousas.net>:
>>>> 
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I've read recently a mail about an apache repository for non
>>>>>  public releases at http://cvs.apache.org/repository/, not 
>>>>> mirrored to ibiblio, that we (maven-plugins developers) don't
>>>>>  use.
>>>>> 
>>>>> I suggest setting it up so we can deploy SNAPSHOT versions 
>>>>> there, avoiding user's need to compile sources.
>>>>> 
>>>>> The mail is here 
>>>>> http://article.gmane.org/gmane.comp.jakarta.commons.devel/39469
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 

-- 
Mark Diggory
Software Developer
Harvard MIT Data Center
http://www.hmdc.harvard.edu

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: ASF Repository Release issues (was Re: Repository for snapshots)

[Leo Simons <ls...@jicarilla.org>]

Mark R. Diggory wrote:
> I'm going to cc this discussion onto infrastructure. I think it is a
> valid topic which we need to address there in more detail.

You seem right on target with your e-mails. At least your understanding 
of the situation matches mine :-D. In review...

1) we don't want "snapshots" available through www.apache.org/dist/ nor 
the mirrors of that location. We push only official, versioned, 
pmc-endorsed, signed files to our mirrors, and everything published 
through www.apache.org/dist goes to our mirrors. I think that rule is 
not up for debate (nor do I see a reason to debate it).

2) The ASF does have several machines available to us, and we're still 
figuring out the optimal way to use each of those machines. Suffice to 
say, there is hardware, bandwidth and disk space, and infrastructure is 
here to facilitate the needs of other ASF projects. That said, 
infrastructure also has other concerns to take into account (like 
security), but in principle the goal is to help you meet your goal.

3) Having a distribution-setup-per-project does not scale and is a 
maintainance nightmare. Hence there is a need for multiple projects to 
co-operate on a common setup.

4) The ASF is not interested in providing redistribution of non-ASF 
files (ie performing the role that ibiblio's maven repository has been 
serving so well lately). I've heard several reasons for that; the most 
important one is probably that its stretching the stated goals of the 
foundation by quite a bit ("ASF is in the software creation business, 
not the software distribution business").

5) We do want snapshot builds available for "developer" use. The old 
(current) recommended model for this is that people push these to either 
their own public_html directory or cvs.apache.org/dist. IIRC, 
historically, use of home directories for this was discouraged because 
of disk space limits (the cvs machine was a seperate one), but that's no 
longer a problem.

6) A while back, repository@apache.org was established as a 
"subcommittee" to deal with this whole repository topic and figure out 
these complex issues (the "new" model), and decide on what to do. I'm 
not sure what happened there and if a consensus was ever reached, but 
its probably the most appropriate forum for achieving it.

Just explain what you need, why you need it, and keep all the above 
context in mind (as I think you already do). Please work on 
repository@apache.org to gain a consensus about that stuff. Then 
approach infrastructure@ with some rough info on what you need (ie 
number of GB, expected bandwidth, seperate unix group or other kind of 
permission scheme, etc etc), and we can make something happen. 
Especially if there's volunteers to help out :-D

A final note is that I've been also dabbling in a related area using the 
gump box: automated nightly builds. Info on that is @ 
http://wiki.apache.org/gump/NightlyBuilds.


Now some loose comments just to make sure this is all clear...
> Jason van Zyl wrote:
>> On Sat, 2004-07-17 at 21:03, Brett Porter wrote:
>>
>>> I don't know very much about cvs.apache.org/repository.
>>>
>>> What I was suggesting was that snapshots that we want to keep should 
>>> go to /dist/java-repository (which is what happens now).

What goes into /dist needs to be versioned, signed, pmc-endorsed, and be 
intented to live "forever" (ie several years). This is the agreement 
with the ASF mirrors. Please don't just push snapshots there.

> Eventually we should
> actually have a ASF Repository structure that all Apache releases go

well, we do have that already (the mirror guidelines). Its just that it 
doesn't work that well for nightly builds, snapshots, etc.

> Most projects out there do maintain major version releases in dist, and
> if they are removed from the projects releases directory, its upto the
> mirror to decide its cleanup strategy in rsync, this means some mirrors
> will not delete copies, while others will. I beleive archive.apache.org
> is meant to be such a mirror that maintains all the content that was
> ever in the www.apache.org/dist directory.

yep.

> As such, IMHO, the restictions placed on dist are to reduce the
> archiving of any unofficial releases. So, IMHO placing any "dated build"
> releases into the ASF Repository violates the requirements because they
> will be "archived" and they are not a full release.

yep.

> I recognize its almost impossible for Infrustructure to enforce this
> policy;

Infrastructure won't. Any problems will just get pushed back to the 
relevant PMC for resolution. ASF projects are self-managing and hence 
responsible for this enforcement :-D

> for example; managing the release of every single maven plugin
> within Infrustructure is probibly not in their interest. What is in
> their interest is being able to maintain a secure release directory with
> properly signed releases.

yes. Everyone's interest, in fact :-D

> I'm convinced we are at a point where
> Infrastructure needs to reflect on the current release strategy and
> attempt to open it up so that we can manage more complex release
> scenarios. We also need to establish a course of action to migrate the
> directory structures to that structure defined by the ASF Repository Spec.

seems like it. I'll point out most of the non-java projects (ie httpd) 
have no particular reason to want to migrate (no incentive for them). So 
"spec backward compatibility" is something to consider as 
well...consistent structure is nice.

> The battle here for us is to maintain "group cohesion" between all the
> Apache projects in terms of where and how to release content, no matter
> how painful interaction can become. So, pushing content off to a
> different location isn't very beneficial in this regard. It may be
> tedious and frustrating to deal with, but we need to maintain such
> interaction.

That's indeed optimal. Doesn't mean that in the meantime "stopgap" 
solutions are out of the question. IE just setting up 
/www/cvs.apache.org/repository and giving everyone in apsite write 
permissions and syncing that to ibiblio is not bad per se.

> What this means to me is that we have a few possible solutions for the
> future direction of ASF Repository:
> 
> 1.) do not allow any interim content into www/dist. Interm content gets
> placed under cvs.apache.org/repository. This gives us a means to allow
> testing of interim content while not having all the mirrors (including
> ibibilio and the archive) get flooded with it.

I think it is very hard to change the relationship between the ASF and 
all its mirrors, ie this is a hard-to-change rule. Some smart technical 
solution is probably a lot easier.

> 2.) Establish a means to filter what gets into the archive (and possibly
> the mirrors). Store all content under one repository location (in dist).
> This is allot more complex and will require effort across multiple
> projects and infrastructure to establish as a practice. (I'm not
> convinced this will be possible).

well I can imagine that the maven release/artifact/dist plugins 
understand when something is a "release" as opposed to a snapshot or 
temporary build easily enough. Excalibur is currently just parsing the 
<version/> and deciding the publication path based on that...

> 3.) Establish the ASF Repository independent of the www/dist directory
> and manage all the content within it, providing our own mechanisms for
> archiving and allowing interim builds. Initial structure will be "Maven"
> like, migrating however, to the ASF Repository Specification. The
> problem is that this fractures the community into those that want to
> release in the old directory structure and those that want to release in
> the Repository.

Another downside is that releases of people using the "new" setup won't 
be mirrored. Bandwidth may be cheap, but everyone downloading maven 1.0 
from the ASF machines instead of a local mirror still is not (both in 
terms of money and efficient network usage).

> So, theres allot of discussion and voting that needs to continue on this
> subject to determine the best course of action, I think that the current
> compromise is not an effective solution for the long term, It reflects my
> initial efforts to establish a repository for ASF projects and now that
> we have our footing we need more member involvement and decision making
> to establish the proper direction for this to go into. This kind of
> growth is hard, changing the way a project releases is a project in and
> of itself.

All the more thanks go out to all the people trying! :-D

>>>> Do you know what the lifespan is for snapshots stored in 
>>>> cvs.apache.org/repository

There is no "rule" on that from infrastructure, nor do I believe there's 
a process that cleans it out...


cheers,


- LSD


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: ASF Repository Release issues (was Re: Repository for snapshots)

[Joshua Slive <jo...@slive.ca>]

On Sun, 18 Jul 2004, Mark R. Diggory wrote:
> 1.) do not allow any interim content into www/dist. Interm content gets
> placed under cvs.apache.org/repository. This gives us a means to allow
> testing of interim content while not having all the mirrors (including
> ibibilio and the archive) get flooded with it.

I am not intimately familiar with the maven way of doing things.  (I am 
intimately familiar with the apache mirror system.)  In order to 
understand the problem here, perhaps someone can explain exactly what is 
the problem with number 1 above.  This is the current status-quo and seems 
to work fine for most projects.

There are several good reasons we don't want snapshot releases in the main 
dist directory:

1) Communication: An apache release should be something the developers 
stand behind as quality software, not an arbitrary, untested snapshot of 
cvs.  If projects (with the approval of their PMC) want to make beta 
releases and place them in the main dist directory, that is fine.  But 
users should know that the stuff they grab from www.apache.org/dist/ has 
some level of quality.

2) Bandwidth: Anything placed under www.apache.org/dist/ gets an automatic 
500 downloads in its first month (and around 300 in its first day) through 
rsyncs from mirrors.  Putting very-short-lived stuff on here could result 
in excessive bandwith usage, which is exactly the opposite of what the 
mirrors are supposed to provide.

3) Archiving: Our archives are made up of everything that has lived on 
www.apache.org/dist/.  Putting very-short-lived stuff there will result in 
bloated and irrelevant archives.

Joshua.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org