Moving dist to svnpubsub

[sebb <se...@gmail.com>]

By next year we need to start publishing releases using svnpubsub.

This involves uploading release candidates to the staging area

https://dist.apache.org/repos/dist/dev/jmeter/

When a vote succeeds, the files can be renamed to the live area here:

https://dist.apache.org/repos/dist/release/jmeter/

Files added to the dist/release area are automatically published to
the ASF mirror system.

Note: Maven files will still be published using Nexus.

I've set up the jmeter directories, and have asked infra [1] for
commit messages to be sent to commits@

I'm also working on updating build.xml to automate the process
(similarly to maven_upload).
There will be some new targets:
- upload archives, sigs and hashes for a specific release to the staging area.
- move files from staging to release area (if vote succeeds)
- delete files from staging area (if vote fails)

The idea is to use the command-line utility "svnmucc" which allows
multiple operations to be done as a single commit transaction.
This has the advantage that all the files appear in the same commit
message; also it should not be possible to forget to upload a sig or
hash.
It also has a "put" function which is ideal for uploading new files -
no need to create a local workspace first.

I've got a basic upload working using the JMeter test area [2]. (You
may have noticed some recent commits of sigs and hashes).

At present the file names are generated by scanning the dist/ area.
That relies on exactly the correct files being present, so I am
considering using a preset list of files, and reporting any
discrepancies (additions as well as omissions)

At some point we will need to move the site to svnpubsub as well, but
that is for a separate thread at a later date.

[1] https://issues.apache.org/jira/browse/INFRA-5557
[2] https://svn.apache.org/repos/asf/jmeter/test/

Re: Moving dist to svnpubsub

[sebb <se...@gmail.com>]

On 27 November 2012 11:38, sebb <se...@gmail.com> wrote:
> On 26 November 2012 01:57, sebb <se...@gmail.com> wrote:
>> On 23 November 2012 00:37, sebb <se...@gmail.com> wrote:
>>> By next year we need to start publishing releases using svnpubsub.
>>>
>>> This involves uploading release candidates to the staging area
>>>
>>> https://dist.apache.org/repos/dist/dev/jmeter/
>>>
>>> When a vote succeeds, the files can be renamed to the live area here:
>>>
>>> https://dist.apache.org/repos/dist/release/jmeter/
>>>
>>> Files added to the dist/release area are automatically published to
>>> the ASF mirror system.
>>>
>>> Note: Maven files will still be published using Nexus.
>>>
>>> I've set up the jmeter directories, and have asked infra [1] for
>>> commit messages to be sent to commits@
>>
>> This has now been done.
>>
>> I've also uploaded the files to the svnpusub release area [1] from
>> minotaur [2] and asked Infra [3] to make the svnpubsub live.
>>
>> [1] https://dist.apache.org/repos/dist/release/jmeter/
>> [2] http://www.apache.org/dist/jmeter/
>> [3] https://issues.apache.org/jira/browse/INFRA-5564
>>
>>> I'm also working on updating build.xml to automate the process
>>> (similarly to maven_upload).
>>> There will be some new targets:
>>> - upload archives, sigs and hashes for a specific release to the staging area.
>>> - move files from staging to release area (if vote succeeds)
>>> - delete files from staging area (if vote fails)
>>>
>>> The idea is to use the command-line utility "svnmucc" which allows
>>> multiple operations to be done as a single commit transaction.
>>> This has the advantage that all the files appear in the same commit
>>> message; also it should not be possible to forget to upload a sig or
>>> hash.
>>> It also has a "put" function which is ideal for uploading new files -
>>> no need to create a local workspace first.
>>
>> [Though svn import also allows this it is not quite as flexible]
>>
>>> I've got a basic upload working using the JMeter test area [2]. (You
>>> may have noticed some recent commits of sigs and hashes).
>>>
>>> At present the file names are generated by scanning the dist/ area.
>>> That relies on exactly the correct files being present, so I am
>>> considering using a preset list of files, and reporting any
>>> discrepancies (additions as well as omissions)
>>
>> I decided to just use a preset list of files.
>> If any are missing, the upload will detect them.
>>
>> I've got the basics working, but there are still some details to tweak.
>>
>> I think it would be easiest for the RC voting if the files were
>> uploaded to a single directory named after the RC.
>>
>> For example,
>>
>> https://dist.apache.org/repos/dist/dev/jmeter/jmeter-2.8RC2/
>>
>> If the vote succeeds, the publish script will rename the files to
>> binaries/ or source/ as appropriate.
>
> This is now implemented in the build script.
>
> Once an RC build has been created, the archives/sigs/hashes can be
> uploaded as follows:
>
> $ ant RC_upload -DRC=RCn [-DrepoType=live]
>
> The default repoType at present is "test", which means that the files
> will be uploaded under:
>
> https://svn.apache.org/repos/asf/jmeter/test/
>
> specifically under
>
> https://svn.apache.org/repos/asf/jmeter/test/dev/jmeter/v2.9_RC1 (for example)
>
> For the live repo - i.e. https://dist.apache.org/repos/dist/ - the
> destination would be:
>
> https://dist.apache.org/repos/dist/dev/jmeter/v2.9_RC1
>
> Once the vote has passed, the RC can be published using the following command:
>
> $ ant RC_upload -DRC=RCn [-DrepoType=live]
>
> This will move the files from
>
> <repo>/dev/jmeter/v2.9_RC1
>
> to
>
> <repo>/release/jmeter/binaries
> or
> <repo>/release/jmeter/source
>
> as appropriate (_src => source)
>
> Note that as soon as the files have been committed to the live repo
> under the release/ tree, they will be copied to the ASF mirrors.
> [It will still take a day or so to propagate to external mirrors]
>
> This is why the repoType defaults to "test" rather than "live".
>
> The RC_publish target does not currently delete the staging directory, i.e.
>
> <repo>/dev/jmeter/v2.9_RC1
>
> This will have to be done manually (using SVN) as this will allow one
> to check that folder is empty before deletion.
> Maybe this can be automated later - it would be ideal if the script
> failed if the staging directory was not empty at the end of the moves.
> [But I think directory deletion currently deletes all content].
>
> Manual deletion is also required when an RC vote fails.
> Failed RC uploads should be deleted before the successful publish to
> avoid possible incorrect publish.
>
> Likewise, there is currently no target for deleting old release from
> the release directories.
> It would be easy enough to add a target to create an svnmucc script,
> but I'm a bit wary of automatically invoking it.
>
> The svnmucc scripts are processed as a single transaction, so if any
> command fails, the entire script will fail.
> This is intentional, but it does mean that one cannot re-run an
> RC_upload without removing the previous staging directory first.
>
> We may need to tweak the process later, but I think (hope!) it is
> workable at present.

Just realised that the svnmucc command relies on using stored
credentials, which is not ideal.
[Sorry, forgot I had used stored credentials for testing]

I'll need to update the script to allow the SVN user name and password
to be specified on the command-line.
AFAICT it's not possible for the exec'ed application to read from the
console; it just hangs.

>>> At some point we will need to move the site to svnpubsub as well, but
>>> that is for a separate thread at a later date.
>>>
>>> [1] https://issues.apache.org/jira/browse/INFRA-5557
>>> [2] https://svn.apache.org/repos/asf/jmeter/test/

Re: Moving dist to svnpubsub

[sebb <se...@gmail.com>]

On 26 November 2012 01:57, sebb <se...@gmail.com> wrote:
> On 23 November 2012 00:37, sebb <se...@gmail.com> wrote:
>> By next year we need to start publishing releases using svnpubsub.
>>
>> This involves uploading release candidates to the staging area
>>
>> https://dist.apache.org/repos/dist/dev/jmeter/
>>
>> When a vote succeeds, the files can be renamed to the live area here:
>>
>> https://dist.apache.org/repos/dist/release/jmeter/
>>
>> Files added to the dist/release area are automatically published to
>> the ASF mirror system.
>>
>> Note: Maven files will still be published using Nexus.
>>
>> I've set up the jmeter directories, and have asked infra [1] for
>> commit messages to be sent to commits@
>
> This has now been done.
>
> I've also uploaded the files to the svnpusub release area [1] from
> minotaur [2] and asked Infra [3] to make the svnpubsub live.
>
> [1] https://dist.apache.org/repos/dist/release/jmeter/
> [2] http://www.apache.org/dist/jmeter/
> [3] https://issues.apache.org/jira/browse/INFRA-5564
>
>> I'm also working on updating build.xml to automate the process
>> (similarly to maven_upload).
>> There will be some new targets:
>> - upload archives, sigs and hashes for a specific release to the staging area.
>> - move files from staging to release area (if vote succeeds)
>> - delete files from staging area (if vote fails)
>>
>> The idea is to use the command-line utility "svnmucc" which allows
>> multiple operations to be done as a single commit transaction.
>> This has the advantage that all the files appear in the same commit
>> message; also it should not be possible to forget to upload a sig or
>> hash.
>> It also has a "put" function which is ideal for uploading new files -
>> no need to create a local workspace first.
>
> [Though svn import also allows this it is not quite as flexible]
>
>> I've got a basic upload working using the JMeter test area [2]. (You
>> may have noticed some recent commits of sigs and hashes).
>>
>> At present the file names are generated by scanning the dist/ area.
>> That relies on exactly the correct files being present, so I am
>> considering using a preset list of files, and reporting any
>> discrepancies (additions as well as omissions)
>
> I decided to just use a preset list of files.
> If any are missing, the upload will detect them.
>
> I've got the basics working, but there are still some details to tweak.
>
> I think it would be easiest for the RC voting if the files were
> uploaded to a single directory named after the RC.
>
> For example,
>
> https://dist.apache.org/repos/dist/dev/jmeter/jmeter-2.8RC2/
>
> If the vote succeeds, the publish script will rename the files to
> binaries/ or source/ as appropriate.

This is now implemented in the build script.

Once an RC build has been created, the archives/sigs/hashes can be
uploaded as follows:

$ ant RC_upload -DRC=RCn [-DrepoType=live]

The default repoType at present is "test", which means that the files
will be uploaded under:

https://svn.apache.org/repos/asf/jmeter/test/

specifically under

https://svn.apache.org/repos/asf/jmeter/test/dev/jmeter/v2.9_RC1 (for example)

For the live repo - i.e. https://dist.apache.org/repos/dist/ - the
destination would be:

https://dist.apache.org/repos/dist/dev/jmeter/v2.9_RC1

Once the vote has passed, the RC can be published using the following command:

$ ant RC_upload -DRC=RCn [-DrepoType=live]

This will move the files from

<repo>/dev/jmeter/v2.9_RC1

to

<repo>/release/jmeter/binaries
or
<repo>/release/jmeter/source

as appropriate (_src => source)

Note that as soon as the files have been committed to the live repo
under the release/ tree, they will be copied to the ASF mirrors.
[It will still take a day or so to propagate to external mirrors]

This is why the repoType defaults to "test" rather than "live".

The RC_publish target does not currently delete the staging directory, i.e.

<repo>/dev/jmeter/v2.9_RC1

This will have to be done manually (using SVN) as this will allow one
to check that folder is empty before deletion.
Maybe this can be automated later - it would be ideal if the script
failed if the staging directory was not empty at the end of the moves.
[But I think directory deletion currently deletes all content].

Manual deletion is also required when an RC vote fails.
Failed RC uploads should be deleted before the successful publish to
avoid possible incorrect publish.

Likewise, there is currently no target for deleting old release from
the release directories.
It would be easy enough to add a target to create an svnmucc script,
but I'm a bit wary of automatically invoking it.

The svnmucc scripts are processed as a single transaction, so if any
command fails, the entire script will fail.
This is intentional, but it does mean that one cannot re-run an
RC_upload without removing the previous staging directory first.

We may need to tweak the process later, but I think (hope!) it is
workable at present.

>> At some point we will need to move the site to svnpubsub as well, but
>> that is for a separate thread at a later date.
>>
>> [1] https://issues.apache.org/jira/browse/INFRA-5557
>> [2] https://svn.apache.org/repos/asf/jmeter/test/

Re: Moving dist to svnpubsub

[sebb <se...@gmail.com>]

On 23 November 2012 00:37, sebb <se...@gmail.com> wrote:
> By next year we need to start publishing releases using svnpubsub.
>
> This involves uploading release candidates to the staging area
>
> https://dist.apache.org/repos/dist/dev/jmeter/
>
> When a vote succeeds, the files can be renamed to the live area here:
>
> https://dist.apache.org/repos/dist/release/jmeter/
>
> Files added to the dist/release area are automatically published to
> the ASF mirror system.
>
> Note: Maven files will still be published using Nexus.
>
> I've set up the jmeter directories, and have asked infra [1] for
> commit messages to be sent to commits@

This has now been done.

I've also uploaded the files to the svnpusub release area [1] from
minotaur [2] and asked Infra [3] to make the svnpubsub live.

[1] https://dist.apache.org/repos/dist/release/jmeter/
[2] http://www.apache.org/dist/jmeter/
[3] https://issues.apache.org/jira/browse/INFRA-5564

> I'm also working on updating build.xml to automate the process
> (similarly to maven_upload).
> There will be some new targets:
> - upload archives, sigs and hashes for a specific release to the staging area.
> - move files from staging to release area (if vote succeeds)
> - delete files from staging area (if vote fails)
>
> The idea is to use the command-line utility "svnmucc" which allows
> multiple operations to be done as a single commit transaction.
> This has the advantage that all the files appear in the same commit
> message; also it should not be possible to forget to upload a sig or
> hash.
> It also has a "put" function which is ideal for uploading new files -
> no need to create a local workspace first.

[Though svn import also allows this it is not quite as flexible]

> I've got a basic upload working using the JMeter test area [2]. (You
> may have noticed some recent commits of sigs and hashes).
>
> At present the file names are generated by scanning the dist/ area.
> That relies on exactly the correct files being present, so I am
> considering using a preset list of files, and reporting any
> discrepancies (additions as well as omissions)

I decided to just use a preset list of files.
If any are missing, the upload will detect them.

I've got the basics working, but there are still some details to tweak.

I think it would be easiest for the RC voting if the files were
uploaded to a single directory named after the RC.

For example,

https://dist.apache.org/repos/dist/dev/jmeter/jmeter-2.8RC2/

If the vote succeeds, the publish script will rename the files to
binaries/ or source/ as appropriate.

> At some point we will need to move the site to svnpubsub as well, but
> that is for a separate thread at a later date.
>
> [1] https://issues.apache.org/jira/browse/INFRA-5557
> [2] https://svn.apache.org/repos/asf/jmeter/test/