Re: lotsa files and svnpubsub was: svn commit: r843406

[Brett Porter <br...@apache.org>]

On 02/01/2013, at 11:43 PM, Brett Porter <br...@apache.org> wrote:

> 
> On 02/01/2013, at 10:45 AM, Hervé BOUTEMY <he...@free.fr> wrote:
> 
>> Le lundi 24 décembre 2012 14:17:02 Brett Porter a écrit :
>>> I had the same feeling pushing up Continuum's Maven site recently...
>>> 
>>> On 23/12/2012, at 9:36 AM, Olivier Lamy <ol...@apache.org> wrote:
>>>> 2012/12/22 Kristian Rosenvold <kr...@zenior.no>:
>>>>> Svnsubpub is just ridiculously inefficient and we need to do something
>>>>> about it...
>>>> 
>>>> That remember me old time when pushing maven sites to sourceforge tru
>>>> wagon ftp ... :-)
>>> 
>>> Is the main problem the file-by-file nature, or something else (such as the
>>> size)?
>> I tried to measure and find facts about this: my conclusion is that it is the 
>> file-by-file nature
>> exactly like webdav
>> notice that we have really a bunch of generated files: did you realize that 
>> maven-scm site has more files than maven core? that it represents 131 MB, tens 
>> of thousand of files?
> 
> Yes, similar size in Continuum which I've been working on migrating to svnpubsub recently.
> 
>> 
>> IMHO, if we stored zip files of documentations (or tar, the archive format can 
>> be discussed) and httpd could serve their content on the fly like if they were 
>> unzipped, this would be awesome
>> Apparently, coding an lua script could do the job: just need to find somebody 
>> able to do it :)
> 
> I've raised this topic and that thread on #asfinfra and will try and follow up - I know this was flagged by them some time ago.

See my response to your mail over there today - this may already be possible: http://www.apache.org/dev/cmsref.html#generated-docs

But aside from that...

> 
>> 
>> any other idea?
> 
> For now I'm just focusing on reducing the number of changes each site deployment to make it a smaller checkin.
> 
> See: http://continuum.apache.org/development/publishing-site.html
> 
> What I'm doing there is deploying to /docs/latest each time, then copying that to the versioned location on the server-side. That way, the next version is just the diff against the last, not a whole new one.
> 
> There are a few things that cause changes across generation runs:
> - the javadocs generate a timestamp, and this is the biggest culprit as it is the largest number of files. I'm going to add -notimestamp tomorrow and see if I can get sequential deploys to remain unmodified
> - most skins incorporated a timestamp that changes each build (I've removed that in Continuum's skin). The publish date is still there, but it doesn't seem to be a big problem.
> - the dependencies report seems to change between runs

By addressing those 3 things, I got down to either no, or very few, changes between commits (it seems Javadoc doesn't consistently order some elements on the page), e.g.: http://svn.apache.org/r1428378

The dependencies issue was fixed in MPIR-266 (which may need a release). Making the skin timestamp configurable is another task.

I added a few tips to the ASF site: http://www.staging.apache.org/dev/project-site.html#generated

> 
> For other reports the change more frequently, I've stopped publishing them to the main site. There is Sonar for most of it at http://analysis.apache.org/, or you can publish them to a filesystem staging in CI and view them through the workspace.
> 
> With these sorts of improvements, incremental deployments will actually be better than shipping up a large ZIP that gets unpacked, even though it is mostly unchanged.


Does this help?

Cheers,
Brett

--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter
http://twitter.com/brettporter






---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org