You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Greg Stein <gs...@gmail.com> on 2011/07/01 13:42:09 UTC
fetch-all-cws.sh (was: Building a single Hg repository)
On Wed, Jun 29, 2011 at 05:04, Michael Stahl <ms...@openoffice.org> wrote:
>...
> in principle the size of a CWS is on the same order as the master, because
> it's just another HG repository.
>
> but HG supports hardlinks between repositories (in newer versions even on
> win32), so you can "hg clone" the master on the same filesystem and then
> pull in the CWS, and it will be _much_ faster and take _much_ less
> additional space
This is the approach that I took. Please look at
tools/dev/fetch-all-cws.sh. Each of these CWS repositories (on Mac OS)
are consuming 600 Mb *minimum*. I've fetched a dozen, and a couple are
over 2 Gb each, and another over 1 Gb. And this is with the clone/pull
technique.
I don't have enough space on my laptop to do a complete trial run. I'm
hoping that somebody can figure out how to reduce the disk footprint,
or determine that we just have to suck it up. And it would be nice to
understand what that target size will be, for all 250 CWS
repositories.
A possible alternative to pulling N repositories, then combining them
in a second step, is to attempt to bring them all into a single
repository, one at a time. This is a little more scary for me, not
knowing Hg, to understand how restartable and repeatable this process
will be in the face of errors. Either starting from scratch, or (I
believe an important feature) if it needs to be resumed after some
minor failure (eg. network failure).
We have a script. It is time to make it work.
Michael: you say that some CWS repositories are useless. If so, then
please update tools/dev/cws-list.txt to comment-out those CWS's with
some explanation for future readers. No need for us to attempt to
process them if they're bogus.
Any thoughts, please?
Cheers,
-g
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Christian Lohmaier <cl...@openoffice.org>.
Hi Greg, *,
On Fri, Jul 1, 2011 at 1:42 PM, Greg Stein <gs...@gmail.com> wrote:
> On Wed, Jun 29, 2011 at 05:04, Michael Stahl <ms...@openoffice.org> wrote:
>>...
>> in principle the size of a CWS is on the same order as the master, because
>> it's just another HG repository.
>>
>> but HG supports hardlinks between repositories (in newer versions even on
>> win32), so you can "hg clone" the master on the same filesystem and then
>> pull in the CWS, and it will be _much_ faster and take _much_ less
>> additional space
> [...]
> I've fetched a dozen, and a couple are
> over 2 Gb each, and another over 1 Gb. And this is with the clone/pull
> technique.
Then something is obviously wrong. Or you're using the wrong
tool/parameters to measure disk-usage. In any case would be helpful
if you stated what cws for example would use up so much more despite
being a clone only.
saying that the local clone+pull uses 2GB or repository-data is hard
to believe, after all the size of a repo-only clone with all cws
pulled in only consumes 2.6GB (all cws=all cws that the tinderbox did
built, this doesn't include cws that are not flagged as public in EIS,
didn't compare how many this would be)
Currently that repo has 134 heads = 133 non-integrated cws/cws with
changes compared to DEV300 (as already integrated cws don't add a new
head, all their changes are included in the DEV300 repo already)
> I don't have enough space on my laptop to do a complete trial run. I'm
> hoping that somebody can figure out how to reduce the disk footprint,
Well, stating the obvious: You must not cross filesystems/disk
partitions otherwise you don't get hardlinks. And you should not use
an ancient version of mercurial (but that's true for every vcs)
> or determine that we just have to suck it up. And it would be nice to
> understand what that target size will be, for all 250 CWS
> repositories.
See above much less than 5GB
> A possible alternative to pulling N repositories, then combining them
> in a second step, is to attempt to bring them all into a single
> repository, one at a time. This is a little more scary for me, not
> knowing Hg,
Nothing scary at all. (well, scary to git users who are afraid of
having multiple heads - with git this really is a problem, but it's
very different in hg).
The only thing is to keep track which head belongs to what cws - using
bookmarks was suggested, but if you prefer, you can just keep a flat
list of hg id = head revisions of the cws along with your clone.
ciao
Christian
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Rob Weir <ap...@robweir.com>.
On Wed, Jul 6, 2011 at 6:41 PM, Simon Phipps <si...@webmink.com> wrote:
>
> On 6 Jul 2011, at 23:19, Rob Weir wrote:
>
>> I think we're transitioning from a formally coordinated model led by a
>> corporation to a model that will have less formal coordination
>> structures. This is necessary to grow the project in the absence of
>> corporate control. But it will take some time to get used to it.
>
> I've always assumed that the extremely careful approach the development team was using was more a function of the size, complexity and intra-dependency of the code than of corporate control. It is amazing how a small, obvious change in one part of OOo can result in a nasty intermittent fault in another part...
>
I've heard rumors of large C/C++ software applications where this is
not true, but I have yet to see one.
-Rob
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
On 07.07.2011 00:41, Simon Phipps wrote:
>
> On 6 Jul 2011, at 23:19, Rob Weir wrote:
>
>> I think we're transitioning from a formally coordinated model led
>> by a corporation to a model that will have less formal
>> coordination structures. This is necessary to grow the project in
>> the absence of corporate control. But it will take some time to
>> get used to it.
>
> I've always assumed that the extremely careful approach the
> development team was using was more a function of the size,
> complexity and intra-dependency of the code than of corporate
> control. It is amazing how a small, obvious change in one part of OOo
> can result in a nasty intermittent fault in another part...
Indeed it was not "corporate control", it was carefulness, though often
exaggerated. Call it "overprotection".
Regards,
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Simon Phipps <si...@webmink.com>.
On 6 Jul 2011, at 23:19, Rob Weir wrote:
> I think we're transitioning from a formally coordinated model led by a
> corporation to a model that will have less formal coordination
> structures. This is necessary to grow the project in the absence of
> corporate control. But it will take some time to get used to it.
I've always assumed that the extremely careful approach the development team was using was more a function of the size, complexity and intra-dependency of the code than of corporate control. It is amazing how a small, obvious change in one part of OOo can result in a nasty intermittent fault in another part...
S.
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Rob Weir <ap...@robweir.com>.
On Wed, Jul 6, 2011 at 5:59 PM, Mathias Bauer <Ma...@gmx.net> wrote:
> On 06.07.2011 18:58, Greg Stein wrote:
>
>> The development process that OOo used to use, as I understand it,
>> looks incredibly heavyweight and slow moving. At Apache, you commit
>> your changes. If you have a large-ish feature you're unsure about,
>> then discuss it on the mailing list, and (maybe) go start a branch. In
>> many cases, it is possibly to develop even large features on trunk
>> because you can "hide" it or make it have near-zero impact on the main
>> trunk code.
>
> There's always a trade-off. I agree that for many code changes the
> development process of OOo was just too heavy. But OTOH it is a well
> known fact that bugs are best found as early as possible. In a large and
> complex code base with a lot of parallel work it happens too easy that
> ugly bugs slip in and cause a lot of trouble for other developers or
> even for users. Fixing them much later, when more code has been added
> and the developer already has moved his mind to something else is much
> more complicated. So a balance is necessary.
>
And different approaches might be used in different parts of the
cycle, or on different parts of the project. For example, we might
declare a Commit-Then-Review policy except after beta, when we turn to
Review-Then-Commit. Or we might be C-T-R for all documentation files
at all times (since they are trivial to review via diff's) while R-T-C
is always the policy for a specific list of shared modules, where
breakage would have a project-wide impact.
But generally coordination has cost. If you are a very small group,
you don't need much formal coordination. But as you get larger you
get to the point where the "friction" from errors outweighs the cost
of coordination. But if you get even larger, the cost of coordination
is even greater, and unless you have high discipline coordination is
hard to achieve, outside of authoritative structures like corporations
or armies, At some scale, nothing but loose coordination will scale.
At the far end you have free markets.
I think we're transitioning from a formally coordinated model led by a
corporation to a model that will have less formal coordination
structures. This is necessary to grow the project in the absence of
corporate control. But it will take some time to get used to it.
> Perhaps looking on other large projects (Firefox, Linux Kernel ...) is a
> good idea.
>
> Regards,
> Mathias
>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
On 06.07.2011 18:58, Greg Stein wrote:
> The development process that OOo used to use, as I understand it,
> looks incredibly heavyweight and slow moving. At Apache, you commit
> your changes. If you have a large-ish feature you're unsure about,
> then discuss it on the mailing list, and (maybe) go start a branch. In
> many cases, it is possibly to develop even large features on trunk
> because you can "hide" it or make it have near-zero impact on the main
> trunk code.
There's always a trade-off. I agree that for many code changes the
development process of OOo was just too heavy. But OTOH it is a well
known fact that bugs are best found as early as possible. In a large and
complex code base with a lot of parallel work it happens too easy that
ugly bugs slip in and cause a lot of trouble for other developers or
even for users. Fixing them much later, when more code has been added
and the developer already has moved his mind to something else is much
more complicated. So a balance is necessary.
Perhaps looking on other large projects (Firefox, Linux Kernel ...) is a
good idea.
Regards,
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Greg Stein <gs...@gmail.com>.
On Sat, Jul 2, 2011 at 12:45, Michael Stahl <ms...@openoffice.org> wrote:
>...
>> Once we have a better picture of what is ready or not cws-list.txt needs
>> to be updated.
>
> i think the goal is to migrate all open CWSes, because perhaps there's still
> something useful in there, and we don't have time to investigate every CWS
> now.
Right. Migrating them all into Subversion for later review is best. We
can always 'rm' the branch if the CWS is determined to be useless.
>...
> just tried it out, and it took 80 minutes to produce a 2.6 GB repo with 178
> bookmarks.
> indeed a lot of bookmarks are duplicated, as described above.
>
> tweaked it a bit, see attachment.
You have commit privs. Apply your fixes right away. We don't need to
review your patch first. Remember: Commit-Then-Review. This will get
things done a lot faster.
The development process that OOo used to use, as I understand it,
looks incredibly heavyweight and slow moving. At Apache, you commit
your changes. If you have a large-ish feature you're unsure about,
then discuss it on the mailing list, and (maybe) go start a branch. In
many cases, it is possibly to develop even large features on trunk
because you can "hide" it or make it have near-zero impact on the main
trunk code.
Cheers,
-g
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Michael Stahl <ms...@openoffice.org>.
On 01.07.2011 18:47, Herbert Duerr wrote:
> On 01.07.2011 13:42, Greg Stein wrote:
>> [...] Please look at
>> tools/dev/fetch-all-cws.sh. Each of these CWS repositories (on Mac OS)
>> are consuming 600 Mb *minimum*. I've fetched a dozen, and a couple are
>> over 2 Gb each, and another over 1 Gb. And this is with the clone/pull
>> technique.
>
> Because of the disk space demands I think an approach with one
> repository with one branch per CWS would be better for the initial
> import. In hg there are more opportunities for this approach (using
> NamedBranched, UnnamedBranches, MultipleHeads, LocalBranches or
> Bookmarks) than I as git-fan would care to know. Please see the attached
> file where I hacked together a script that imports the CWSs into one
> repository with multiple bookmarks.
of course, mirroring all the separate repos is only an intermediate
step, with the goal that constructing the single repo doesn't need to go
over the slow net.
>> I don't have enough space on my laptop to do a complete trial run. I'm
>> hoping that somebody can figure out how to reduce the disk footprint,
>> or determine that we just have to suck it up. And it would be nice to
>> understand what that target size will be, for all 250 CWS
>> repositories.
>
> As Michael mentioned it's much less than the 250, as only about 15
> CWSses (see http://goo.gl/gczAH for details) are marked as fully tested
> and but not-yet integrated.
> From these the ones targeted at the stabilization branch (calc67,
> calc68, ooo34gsl01, ooo34gsl02, writerfilter10, native373, jl167) are
> more important than the ones targeted for trunk (sb140, sb143, hsqldb19,
> hr77, ause131, sd2gbuild).
>
> There are a few more very good CWSses which were not yet officially
> approved in the old OOo system. E.g. CWS aw080 Armin mentioned. If Armin
> can confirm that this CWS is ready we should pull it in too.
>
> Once we have a better picture of what is ready or not cws-list.txt needs
> to be updated.
i think the goal is to migrate all open CWSes, because perhaps there's
still something useful in there, and we don't have time to investigate
every CWS now.
>> We have a script. It is time to make it work.
>
> Please see the attached file. I'm afraid to check it in as the related
> single-hg.sh is not yet updated accordingly and also because I'm more of
> a HG victim than a HG power-user ;-)
>
> Please note that you either have to use HG>=1.8 or enable its bookmark
> extension before you run the attached script.
i've not used bookmarks or anything like that either, but it looks like
it could work, except i think we may need to prevent it from creating
wrong bookmarks in case no changes are to be pulled (it seems in this
case, the hg bookmark -r tip would create a bookmark $cws which points
at the previously pulled cws, instead of doing nothing; "tip" is always
the head that was pulled last into the repo).
just tried it out, and it took 80 minutes to produce a 2.6 GB repo with
178 bookmarks.
indeed a lot of bookmarks are duplicated, as described above.
tweaked it a bit, see attachment.
regards,
michael
--
"Simply, I'd say that porting [Linux] is impossible."
-- Linus Torvalds (26. 08. 1991)
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Armin Le Grand <Ar...@me.com>.
Am 01.07.2011 18:47, schrieb Herbert Duerr:
> On 01.07.2011 13:42, Greg Stein wrote:
>> [...] Please look at
>> tools/dev/fetch-all-cws.sh. Each of these CWS repositories (on Mac OS)
>> are consuming 600 Mb *minimum*. I've fetched a dozen, and a couple are
>> over 2 Gb each, and another over 1 Gb. And this is with the clone/pull
>> technique.
>
> Because of the disk space demands I think an approach with one
> repository with one branch per CWS would be better for the initial
> import. In hg there are more opportunities for this approach (using
> NamedBranched, UnnamedBranches, MultipleHeads, LocalBranches or
> Bookmarks) than I as git-fan would care to know. Please see the attached
> file where I hacked together a script that imports the CWSs into one
> repository with multiple bookmarks.
>
>> I don't have enough space on my laptop to do a complete trial run. I'm
>> hoping that somebody can figure out how to reduce the disk footprint,
>> or determine that we just have to suck it up. And it would be nice to
>> understand what that target size will be, for all 250 CWS
>> repositories.
>
> As Michael mentioned it's much less than the 250, as only about 15
> CWSses (see http://goo.gl/gczAH for details) are marked as fully tested
> and but not-yet integrated.
> From these the ones targeted at the stabilization branch (calc67,
> calc68, ooo34gsl01, ooo34gsl02, writerfilter10, native373, jl167) are
> more important than the ones targeted for trunk (sb140, sb143, hsqldb19,
> hr77, ause131, sd2gbuild).
>
> There are a few more very good CWSses which were not yet officially
> approved in the old OOo system. E.g. CWS aw080 Armin mentioned. If Armin
> can confirm that this CWS is ready we should pull it in too.
Well, it will need some more work, but it is ready in the sense that it
is on DEV300 m106 and that I am pushing to the old locations all the
time. Main point is to get over to this dev line so that I can resync to
the trunc here locally ASAP instead of continuing work with what will be
an outdated rep soon.
2nd point is to get into the transition from the licenses ASAP, too. I
can continue (and I do) to work on DEV300 codeline, but as soon as we
have a running build here I opt for migrating.
>
> Once we have a better picture of what is ready or not cws-list.txt needs
> to be updated.
>
>> We have a script. It is time to make it work.
>
> Please see the attached file. I'm afraid to check it in as the related
> single-hg.sh is not yet updated accordingly and also because I'm more of
> a HG victim than a HG power-user ;-)
>
> Please note that you either have to use HG>=1.8 or enable its bookmark
> extension before you run the attached script.
>
> Best regards,
> Herbert Duerr
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Herbert Duerr <hd...@openoffice.org>.
On 01.07.2011 13:42, Greg Stein wrote:
> [...] Please look at
> tools/dev/fetch-all-cws.sh. Each of these CWS repositories (on Mac OS)
> are consuming 600 Mb *minimum*. I've fetched a dozen, and a couple are
> over 2 Gb each, and another over 1 Gb. And this is with the clone/pull
> technique.
Because of the disk space demands I think an approach with one
repository with one branch per CWS would be better for the initial
import. In hg there are more opportunities for this approach (using
NamedBranched, UnnamedBranches, MultipleHeads, LocalBranches or
Bookmarks) than I as git-fan would care to know. Please see the attached
file where I hacked together a script that imports the CWSs into one
repository with multiple bookmarks.
> I don't have enough space on my laptop to do a complete trial run. I'm
> hoping that somebody can figure out how to reduce the disk footprint,
> or determine that we just have to suck it up. And it would be nice to
> understand what that target size will be, for all 250 CWS
> repositories.
As Michael mentioned it's much less than the 250, as only about 15
CWSses (see http://goo.gl/gczAH for details) are marked as fully tested
and but not-yet integrated.
From these the ones targeted at the stabilization branch (calc67,
calc68, ooo34gsl01, ooo34gsl02, writerfilter10, native373, jl167) are
more important than the ones targeted for trunk (sb140, sb143, hsqldb19,
hr77, ause131, sd2gbuild).
There are a few more very good CWSses which were not yet officially
approved in the old OOo system. E.g. CWS aw080 Armin mentioned. If Armin
can confirm that this CWS is ready we should pull it in too.
Once we have a better picture of what is ready or not cws-list.txt needs
to be updated.
> We have a script. It is time to make it work.
Please see the attached file. I'm afraid to check it in as the related
single-hg.sh is not yet updated accordingly and also because I'm more of
a HG victim than a HG power-user ;-)
Please note that you either have to use HG>=1.8 or enable its bookmark
extension before you run the attached script.
Best regards,
Herbert Duerr
Re: fetch-all-cws.sh
Posted by Michael Stahl <ms...@openoffice.org>.
On 07.07.2011 12:00, Michael Stahl wrote:
> On 06.07.2011 18:35, Herbert Duerr wrote:
> i've started converting yesterday evening, and maybe it'll be finished
> today (on my 3 year old laptop...)
conversion finished after ~75hours; it seems to have slowed down over
time, especially the last 20k commits that include merges changing 40k
files turned from CPU bound to IO bound, and took a long time...
seems to have worked, result is a 5.7GB git repo with 104 branches.
>> I'd clarify step one to "convert OOO340 repo to a bare git repo via
>> hg-fast-export.sh". After all these steps please don't forget git
>> pack-refs and git repack (e.g. with "-a -d -f --window=200
>> --depth=1000") to get a nice and tight repository.
git pack-refs didn't seem to do anything.
git repack with your options ran for 6 hours and shrank the repo to ...
5.7GB.
any idea what i'm doing wrong?
Re: fetch-all-cws.sh
Posted by Michael Stahl <ms...@openoffice.org>.
On 06.07.2011 18:35, Herbert Duerr wrote:
> > there is another tool, a HG extension called hg-git, which can
> > convert HG bookmarks to git branches.
> > http://hg-git.github.com/
>
> Great find! I was already brushing up my python and mercurial internals
> skills to extend hg-fast-export's export_commit() for our one big
> hg-repo with one hg-branch and many hg-bookmarks. I'm glad that cup passed.
>
> > so my current plan is this:
> > 1. convert OOO340 repo to git via hg-fast-export.sh
> > 2. pull all CWSes into OOO340 repo and create bookmarks
> > 3. use hg-git to push all of them into the converted git repo
>
> Sounds good!
unfortunately it won't work :(
problem is that hg-fast-export.sh and hg-git don't quite agree what a
converted git repo should look like.
i've actually tried it out with a trivial repo and 2 branches and
surprisingly it actually worked, but then i tried it on a real repo (the
hg-git one), just take some arbitrary changeset and convert up to that
with hg-fast-export.sh, then hg-git push of the rest fails...
next thing i tried is to convert the whole thing via hg-git push, but
unfortunately they weren't lying when they used the word "slow".
after a couple of hours it had converted one percent of the changesets,
and the progress predicted an ETA of >8 days.
so then i've taken a deeper look at the hg-fast-export code, and it
seems surprisingly easy to hack it to do something with HG bookmarks.
basically a HG bookmark points at a single revision, and is thus quite
similar to a git "ref".
the hg-fast-export writes a header for every changeset, with a branch
name: "commit refs/heads/$branchname"
so i'm detecting all the HG heads that are marked by bookmarks, and just
use the bookmark name as the branch name.
also, a bookmark for the head that corresponds to OOO340 is necessary.
then invoke it like this:
mkdir git
cd git
git init
hg-fast-export.sh -r ../hg -M dummy
the "-M dummy" sets the default branch, so all changesets that are _not_
heads end up on the "dummy" branch, and for every head/bookmark a branch
pointing to that head is created.
the "dummy" branch/ref can be deleted after conversion.
i don't understand git very well, but i hope this should work :)
the tool already errors out if there is more than one head without a
branch/bookmark.
the attached patch just adds 3 lines of actual code plus some parameter
shuffling.
(a case that may not be handled properly is if there is a HG branch and
a HG bookmark with the same name, but we don't have HG branches at all...)
i've started converting yesterday evening, and maybe it'll be finished
today (on my 3 year old laptop...)
> I'd clarify step one to "convert OOO340 repo to a bare git repo via
> hg-fast-export.sh". After all these steps please don't forget git
> pack-refs and git repack (e.g. with "-a -d -f --window=200
> --depth=1000") to get a nice and tight repository.
>
> Herbert
Re: fetch-all-cws.sh
Posted by Herbert Duerr <hd...@alice.de>.
>> [...] Speaking of merges in SVN we regularly had the problem that a
>> file was renamed in one CWS and changed in another CWS and the result
>> was that the change got lost. HG or GIT handle this scenario much
>> more gracefully, so there was less risk to do big merges.
>
> indeed, which led to effectively forbidding moving files in SVN...
> i hope SVN has learned to signal a merge conflict for this nowadays?
I found http://subversion.tigris.org/issues/show_bug.cgi?id=898 and it
is in status NEW, but the target milestone is 1.8-consider already, so
there is hope.
> there is another tool, a HG extension called hg-git, which can
> convert HG bookmarks to git branches.
> http://hg-git.github.com/
Great find! I was already brushing up my python and mercurial internals
skills to extend hg-fast-export's export_commit() for our one big
hg-repo with one hg-branch and many hg-bookmarks. I'm glad that cup passed.
> so my current plan is this:
> 1. convert OOO340 repo to git via hg-fast-export.sh
> 2. pull all CWSes into OOO340 repo and create bookmarks
> 3. use hg-git to push all of them into the converted git repo
Sounds good!
I'd clarify step one to "convert OOO340 repo to a bare git repo via
hg-fast-export.sh". After all these steps please don't forget git
pack-refs and git repack (e.g. with "-a -d -f --window=200
--depth=1000") to get a nice and tight repository.
Herbert
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Michael Stahl <ms...@openoffice.org>.
On 06.07.2011 19:12, Greg Stein wrote:
> On Wed, Jul 6, 2011 at 07:50, Michael Stahl<ms...@openoffice.org> wrote:
>> On 05.07.2011 11:16, Herbert Duerr wrote:
>>> If the goal is to just merge the outstanding CWSs into trunk I'd suggest
>>> to stay with hg, merge all good CWSs into trunk and start the apache-ooo
>>> SVN repository from that trunk.
>>>
>>> If the goal is to preserve the trunk and CWSs of the old-OOo project
>>> then the idea to provide it as a read-only git repository is great,
>
> We only have one canonical repository at Apache, and that is
> Subversion. We should not be planning to create a Git repository...
> the read-only Git "repositories" are just mirrors of portions of the
> Subversion repository.
i thought you and/or Mathias were working on the SVN conversion already.
who knows, perhaps there will be trouble converting from HG to SVN, and
going via git as an intermediate step is easier?
it's good to have options :)
regards,
michael
--
"PL/I and Ada started out with all the bloat, were very daunting
languages, and got bad reputations (deservedly). But C++ has shown
that if you slowly bloat up a language over a period of years people
don't seem to mind as much." -- James Hague, 1992
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Jul 6, 2011 at 07:50, Michael Stahl <ms...@openoffice.org> wrote:
> On 05.07.2011 11:16, Herbert Duerr wrote:
>...
>> merged. Speaking of merges in SVN we regularly had the problem that a
>> file was renamed in one CWS and changed in another CWS and the result
>> was that the change got lost. HG or GIT handle this scenario much more
>> gracefully, so there was less risk to do big merges.
>
> indeed, which led to effectively forbidding moving files in SVN...
> i hope SVN has learned to signal a merge conflict for this nowadays?
It sure does. They are called "tree conflicts". In this particular
case, you have an "incoming edit" to a file that doesn't exist. That
will get flagged for conflict resolution.
>...
>> If the goal is to just merge the outstanding CWSs into trunk I'd suggest
>> to stay with hg, merge all good CWSs into trunk and start the apache-ooo
>> SVN repository from that trunk.
>>
>> If the goal is to preserve the trunk and CWSs of the old-OOo project
>> then the idea to provide it as a read-only git repository is great,
We only have one canonical repository at Apache, and that is
Subversion. We should not be planning to create a Git repository...
the read-only Git "repositories" are just mirrors of portions of the
Subversion repository.
>...
Cheers,
-g
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Michael Stahl <ms...@openoffice.org>.
On 05.07.2011 11:16, Herbert Duerr wrote:
> On 05.07.2011, Rob Weir wrote:
>> Is the Hg ==> Git conversion easier/cleaner than the conversion to SVN?
>
> Yes. Having the CWSs in SVN would only increase the size of the
> repository considerably without providing benefits after they get
> merged. Speaking of merges in SVN we regularly had the problem that a
> file was renamed in one CWS and changed in another CWS and the result
> was that the change got lost. HG or GIT handle this scenario much more
> gracefully, so there was less risk to do big merges.
indeed, which led to effectively forbidding moving files in SVN...
i hope SVN has learned to signal a merge conflict for this nowadays?
>> I notice that Apache Infrastructure does support *read-only* Git
>> repositories:
>>
>> http://git.apache.org/
>>
>> Would it be any simpler to move the enter Oracle-hosted project,
>> including all CWS's to a read-only Git, and then create a dump of just
>> the integrated trunk (with history) for the project's SVN repository?
>> Then, we'll have ongoing access to the CWS's in Git until such point
>> as they are ready to be integrated. In other words, treat the Git
>> version as a read-only archive.
>
> If the goal is to just merge the outstanding CWSs into trunk I'd suggest
> to stay with hg, merge all good CWSs into trunk and start the apache-ooo
> SVN repository from that trunk.
>
> If the goal is to preserve the trunk and CWSs of the old-OOo project
> then the idea to provide it as a read-only git repository is great,
> especially since it would work in the current infrastructure. With
> http://repo.or.cz/w/fast-export.git the conversion could be easy.
> Unfortunately that tool hasn't been extended to handle hg-bookmarks yet
> but it handles hg-branches just fine. So if someone more experienced
> with HG can rewrite the latest fetch-all-cws script posted here from
> hg-bookmarks to its hg-branches equivalent we would be only one step
> away from such a "status-quo" repository. I guess the resulting bare
> repo would only consume about 1.5GB for everything.
this is a very interesting idea, so i've tried to figure out how to do
this...
found this article most helpful:
http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial/
unfortunately, (in contrast to HG bookmarks) it doesn't seem to be
possible to retroactively move HG commits onto a HG named branch.
for a HG named branch the branch is part of the commit metadata, so this
would need modifying the existing commits, and i've not found a tool
that can do this (the HG convert extension allows for renaming branches,
but in this case we'd need to move _some_ commits from the default
branch to a CWS branch, and i can't see how that could be done).
there is another tool, a HG extension called hg-git, which can convert
HG bookmarks to git branches.
http://hg-git.github.com/
so my current plan is this:
1. convert OOO340 repo to git via hg-fast-export.sh
2. pull all CWSes into OOO340 repo and create bookmarks
3. use hg-git to push all of them into the converted git repo
it would also be possible to use hg-git to convert the whole HG repo
with bookmarks in one step; don't know which is the better way, and the
hg-git homepage mentions the word "slow" somewhere, while we have
"hg-fast-export.sh", so i'll try this route first :)
> For a more complete history we could then also add the >1000 CWSs from
> the svn period. I found that having the individual instead of the big
> merge-hunks was invaluable for understanding the status of some code.
> Even with these many CWSs the result will probably stay under 2.0GB.
hmm... could be useful :)
--
"The 'sound' banker, alas! is not one who sees danger and avoids it,
but one who, when he is ruined, is ruined in a conventional and
orthodox way along with his fellows so that no one can really
blame him." -- John Maynard Keynes
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Herbert Duerr <hd...@openoffice.org>.
On 05.07.2011, Rob Weir wrote:
> Is the Hg ==> Git conversion easier/cleaner than the conversion to SVN?
Yes. Having the CWSs in SVN would only increase the size of the
repository considerably without providing benefits after they get
merged. Speaking of merges in SVN we regularly had the problem that a
file was renamed in one CWS and changed in another CWS and the result
was that the change got lost. HG or GIT handle this scenario much more
gracefully, so there was less risk to do big merges.
> I notice that Apache Infrastructure does support *read-only* Git repositories:
>
> http://git.apache.org/
>
> Would it be any simpler to move the enter Oracle-hosted project,
> including all CWS's to a read-only Git, and then create a dump of just
> the integrated trunk (with history) for the project's SVN repository?
> Then, we'll have ongoing access to the CWS's in Git until such point
> as they are ready to be integrated. In other words, treat the Git
> version as a read-only archive.
If the goal is to just merge the outstanding CWSs into trunk I'd suggest
to stay with hg, merge all good CWSs into trunk and start the apache-ooo
SVN repository from that trunk.
If the goal is to preserve the trunk and CWSs of the old-OOo project
then the idea to provide it as a read-only git repository is great,
especially since it would work in the current infrastructure. With
http://repo.or.cz/w/fast-export.git the conversion could be easy.
Unfortunately that tool hasn't been extended to handle hg-bookmarks yet
but it handles hg-branches just fine. So if someone more experienced
with HG can rewrite the latest fetch-all-cws script posted here from
hg-bookmarks to its hg-branches equivalent we would be only one step
away from such a "status-quo" repository. I guess the resulting bare
repo would only consume about 1.5GB for everything.
For a more complete history we could then also add the >1000 CWSs from
the svn period. I found that having the individual instead of the big
merge-hunks was invaluable for understanding the status of some code.
Even with these many CWSs the result will probably stay under 2.0GB.
Herbert
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Rob Weir <ap...@robweir.com>.
Is the Hg ==> Git conversion easier/cleaner than the conversion to SVN?
I notice that Apache Infrastructure does support *read-only* Git repositories:
http://git.apache.org/
Would it be any simpler to move the enter Oracle-hosted project,
including all CWS's to a read-only Git, and then create a dump of just
the integrated trunk (with history) for the project's SVN repository?
Then, we'll have ongoing access to the CWS's in Git until such point
as they are ready to be integrated. In other words, treat the Git
version as a read-only archive.
-Rob
On Mon, Jul 4, 2011 at 11:36 AM, Mathias Bauer <Ma...@gmx.net> wrote:
> On 04.07.2011 17:27, Michael Stahl wrote:
>
>> what do you mean by "obsolete"?
>> the "integrated"/"deleted" ones seem to be removed from the server, and
>> i've commented out the "cancelled" ones.
>
> there are two cws that we don't need anymore:
>
> layoutdialogs3 and mh6bc
>
> The first works with code that another cws removed (mbacodecleanup), the
> second is a proof-of-concept-cws for a possible conversion of our build
> system to cmake.
>
> Regards
> Mathias
>
>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
On 04.07.2011 17:27, Michael Stahl wrote:
> what do you mean by "obsolete"?
> the "integrated"/"deleted" ones seem to be removed from the server, and
> i've commented out the "cancelled" ones.
there are two cws that we don't need anymore:
layoutdialogs3 and mh6bc
The first works with code that another cws removed (mbacodecleanup), the
second is a proof-of-concept-cws for a possible conversion of our build
system to cmake.
Regards
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
On 04.07.2011 17:27, Michael Stahl wrote:
> On 04.07.2011 16:21, Mathias Bauer wrote:
>> On 01.07.2011 22:58, Michael Stahl wrote:
>>
>>> i think i wrote that all CWSes as HG repos take ~100 GB, but actually i
>>> now think i remembered wrong and the number was more like ~150 GB.
>>> (i did this originally in 2 steps, and i remembered only the second step...)
>>> (and if it weren't so late now i'd even dig out my external hd and run
>>> du...)
>>
>> I used the "branchmirror" Mercurial extension from Björn to fetch all
>> CWS, based on the OOO340 master repository.
>>
>> There are 185 cws, on a Linux system with hard links their hg repos
>> consume ~80 GB. 66 of the cws are empty, 2 of them are obsolete.
>
> i forgot to mention: on Saturday i've actually run "du" myself (which
> took 20 minutes on the USB disk...), and in fact it's 100GB, so my first
> guess was right.
>
> (guess the fact that yours is smaller is caused by me having used DEV300
> and you OOO340, so some CWSes that are empty for you have content for me)
Indeed I got more empty cws because I checked against OOO340. As this is
the version we want to use, I updated cws-list.txt accordingly.
There is one cws that is not empty in my list, but empty in yours:
ause114. I kept the state from you as the only difference of this cws
against OOO340 are some masterfixes that have been applied after the
branch-off, and they have been applied in similar or comparable way to
OOO340 anyway.
Unfortunately we will see these masterfixes in some cws based on dev300
again, so we need to back them out somehow before we can merge the cws
branches to the trunk.
> take a look at the cws-list.txt in the SVN repo.
> if you see something that shouldn't be migrated, you can comment it out...
Done.
Regards,
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Michael Stahl <ms...@openoffice.org>.
On 04.07.2011 16:21, Mathias Bauer wrote:
> On 01.07.2011 22:58, Michael Stahl wrote:
>
>> i think i wrote that all CWSes as HG repos take ~100 GB, but actually i
>> now think i remembered wrong and the number was more like ~150 GB.
>> (i did this originally in 2 steps, and i remembered only the second step...)
>> (and if it weren't so late now i'd even dig out my external hd and run
>> du...)
>
> I used the "branchmirror" Mercurial extension from Björn to fetch all
> CWS, based on the OOO340 master repository.
>
> There are 185 cws, on a Linux system with hard links their hg repos
> consume ~80 GB. 66 of the cws are empty, 2 of them are obsolete.
i forgot to mention: on Saturday i've actually run "du" myself (which
took 20 minutes on the USB disk...), and in fact it's 100GB, so my first
guess was right.
(guess the fact that yours is smaller is caused by me having used DEV300
and you OOO340, so some CWSes that are empty for you have content for me)
> This leaves us 117 cws to investigate.
>
> In case the list of empty or obsolete cws is important, I could post it
> here.
what do you mean by "obsolete"?
the "integrated"/"deleted" ones seem to be removed from the server, and
i've commented out the "cancelled" ones.
the empty ones can be checked via "hg incoming".
take a look at the cws-list.txt in the SVN repo.
if you see something that shouldn't be migrated, you can comment it out...
--
Life is complex. It has real and imaginary components.
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
On 01.07.2011 22:58, Michael Stahl wrote:
> i think i wrote that all CWSes as HG repos take ~100 GB, but actually i
> now think i remembered wrong and the number was more like ~150 GB.
> (i did this originally in 2 steps, and i remembered only the second step...)
> (and if it weren't so late now i'd even dig out my external hd and run
> du...)
I used the "branchmirror" Mercurial extension from Björn to fetch all
CWS, based on the OOO340 master repository.
There are 185 cws, on a Linux system with hard links their hg repos
consume ~80 GB. 66 of the cws are empty, 2 of them are obsolete.
This leaves us 117 cws to investigate.
In case the list of empty or obsolete cws is important, I could post it
here.
Regards,
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
On 06.07.2011 19:08, Greg Stein wrote:
> On Tue, Jul 5, 2011 at 16:04, Rob Weir <ap...@robweir.com> wrote:
>> On Tue, Jul 5, 2011 at 3:33 PM, Mathias Bauer <Ma...@gmx.net> wrote:
>>> Moin,
>>>
>>> On 05.07.2011 18:14, Mathias Bauer wrote:
>>...
>>> Do we really want to have code in the svn repo that will never be used?
>>> The alternative would be to add cws to svn only after review.
>
> Having code in the repository is fine. It is better to have it there
> and not need it, then to need it and not have it there.
>
> As people have stated, these extra CWSs do not add that much more
> space. And Apache has more than enough disk space. It is not a
> concern.
>
>> Right. That is why I was thinking that maybe we just create an
>> archival copy of the entire repository, including all CWS, and host
>> that as a read only Hg or git instance. Then migrate the trunk to
>> SVN, If there are some CWS that we know are already approved for
>> 3.4, then include those as well.
>
> Apache only has one repository: Subversion.
>
> The read-only Git instance is simply a mirror of the canonical
> Subversion repository.
>
>> That way, if someone does come by, months later, and decide they want
>> to complete work CWS, then they can still clone them and work on them.
>> But then they would need to copy them into a SVN working copy, and
>> merge and commit from there. Obviously, this does complicate things
>> for the future CWS developers. But they are in the best position to
>> stabilize and merge their work.
>
> Haven't we already gone through the whole discussion about CWSs that
> may rename files, and that we want to do that rename within Hg before
> converting to Subversion?
>
> Let's not mess around with multiple repositories. That is just making
> it difficult for us. How is somebody supposed to investigate a CWS
> when it lives in a separate repository? That is just making it harder
> for ourselves.
Yes, you are right. The idea with moving all cws into svn and then work
from there is good enough and trying to make things better is a waste of
time that could even backfire.
So sorry for the noise, let's proceed with the hg->svn migration.
Regards,
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Greg Stein <gs...@gmail.com>.
On Tue, Jul 5, 2011 at 16:04, Rob Weir <ap...@robweir.com> wrote:
> On Tue, Jul 5, 2011 at 3:33 PM, Mathias Bauer <Ma...@gmx.net> wrote:
>> Moin,
>>
>> On 05.07.2011 18:14, Mathias Bauer wrote:
>...
>> Do we really want to have code in the svn repo that will never be used?
>> The alternative would be to add cws to svn only after review.
Having code in the repository is fine. It is better to have it there
and not need it, then to need it and not have it there.
As people have stated, these extra CWSs do not add that much more
space. And Apache has more than enough disk space. It is not a
concern.
> Right. That is why I was thinking that maybe we just create an
> archival copy of the entire repository, including all CWS, and host
> that as a read only Hg or git instance. Then migrate the trunk to
> SVN, If there are some CWS that we know are already approved for
> 3.4, then include those as well.
Apache only has one repository: Subversion.
The read-only Git instance is simply a mirror of the canonical
Subversion repository.
> That way, if someone does come by, months later, and decide they want
> to complete work CWS, then they can still clone them and work on them.
> But then they would need to copy them into a SVN working copy, and
> merge and commit from there. Obviously, this does complicate things
> for the future CWS developers. But they are in the best position to
> stabilize and merge their work.
Haven't we already gone through the whole discussion about CWSs that
may rename files, and that we want to do that rename within Hg before
converting to Subversion?
Let's not mess around with multiple repositories. That is just making
it difficult for us. How is somebody supposed to investigate a CWS
when it lives in a separate repository? That is just making it harder
for ourselves.
-g
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 07/05/2011 10:04 PM, schrieb Rob Weir:
> On Tue, Jul 5, 2011 at 3:33 PM, Mathias Bauer<Ma...@gmx.net> wrote:
>> On 05.07.2011 18:14, Mathias Bauer wrote:
>>
>>> It seems that my memory had fooled me: so if anybody can create an svn
>>> dump file, I will try to recap what we have agreed to so far and have a
>>> look into the conversion. In case anyone else is already at this, please
>>> let me know.
>> Having said that, there's a thought that makes me wonder: we have 117
>> cws with more or less unfinished work. I doubt that we will integrate
>> them all anytime soon, as integration also comprises developing the
>> merged code further until it has sufficient quality for becoming part of
>> the trunk. [Wat is "sufficient" is still undefined - we surely won't
>> continue the overdone QA approval process from the "old" OOo project,
>> but OTOH also shouldn't throw code at the repository at will.]
>>
>> Some of the 117 cws are anbandoned work, others are work in an early
>> state that most probably doesn't make sense to be continued without the
>> developers starting it.
>>
>> Do we really want to have code in the svn repo that will never be used?
>> The alternative would be to add cws to svn only after review.
>>
>
> Right. That is why I was thinking that maybe we just create an
> archival copy of the entire repository, including all CWS, and host
> that as a read only Hg or git instance. Then migrate the trunk to
> SVN, If there are some CWS that we know are already approved for
> 3.4, then include those as well.
>
> That way, if someone does come by, months later, and decide they want
> to complete work CWS, then they can still clone them and work on them.
> But then they would need to copy them into a SVN working copy, and
> merge and commit from there. Obviously, this does complicate things
> for the future CWS developers. But they are in the best position to
> stabilize and merge their work.
OK, to keep the code in a special branch in our new the Apache SVN repo
or in a separated one as archive. For me it seems to be the same: Keep
instead of delete.
Marcus
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by "Pedro F. Giffuni" <gi...@tutopia.com>.
--- On Tue, 7/5/11, Mathias Bauer <Ma...@gmx.net> wrote:
...
>
> I doubt that this will complicate things too much. It
> should be bearable.
>
> I'm interested to read the opinions of those who favored
> the "get everything into svn at once" approach.
>
All the history is valuable to avoid redoing efforts. The
great concern is discerning everything that will be needed
or desired before the Hg server disappears.
Another issue is.. will we be asking Oracle to relicense all
the code in CWS too? That may complicate things.
I think the simple approach you are considering is fine for
now. Without knowing the projects well enough I only know
we will want the hsqldb19 cws.
cheers,
Pedro.
cheers,
Pedro.
> Regards,
> Mathias
>
>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
On 05.07.2011 22:04, Rob Weir wrote:
> On Tue, Jul 5, 2011 at 3:33 PM, Mathias Bauer <Ma...@gmx.net> wrote:
>> Moin,
>>
>> On 05.07.2011 18:14, Mathias Bauer wrote:
>>
>>> It seems that my memory had fooled me: so if anybody can create an svn
>>> dump file, I will try to recap what we have agreed to so far and have a
>>> look into the conversion. In case anyone else is already at this, please
>>> let me know.
>> Having said that, there's a thought that makes me wonder: we have 117
>> cws with more or less unfinished work. I doubt that we will integrate
>> them all anytime soon, as integration also comprises developing the
>> merged code further until it has sufficient quality for becoming part of
>> the trunk. [Wat is "sufficient" is still undefined - we surely won't
>> continue the overdone QA approval process from the "old" OOo project,
>> but OTOH also shouldn't throw code at the repository at will.]
>>
>> Some of the 117 cws are anbandoned work, others are work in an early
>> state that most probably doesn't make sense to be continued without the
>> developers starting it.
>>
>> Do we really want to have code in the svn repo that will never be used?
>> The alternative would be to add cws to svn only after review.
>>
>
> Right. That is why I was thinking that maybe we just create an
> archival copy of the entire repository, including all CWS, and host
> that as a read only Hg or git instance. Then migrate the trunk to
> SVN, If there are some CWS that we know are already approved for
> 3.4, then include those as well.
Yes, that would be my suggestion also. git would fit better here, I assume.
> That way, if someone does come by, months later, and decide they want
> to complete work CWS, then they can still clone them and work on them.
> But then they would need to copy them into a SVN working copy, and
> merge and commit from there. Obviously, this does complicate things
> for the future CWS developers. But they are in the best position to
> stabilize and merge their work.
I doubt that this will complicate things too much. It should be bearable.
I'm interested to read the opinions of those who favored the "get
everything into svn at once" approach.
Regards,
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Rob Weir <ap...@robweir.com>.
On Tue, Jul 5, 2011 at 3:33 PM, Mathias Bauer <Ma...@gmx.net> wrote:
> Moin,
>
> On 05.07.2011 18:14, Mathias Bauer wrote:
>
>> It seems that my memory had fooled me: so if anybody can create an svn
>> dump file, I will try to recap what we have agreed to so far and have a
>> look into the conversion. In case anyone else is already at this, please
>> let me know.
> Having said that, there's a thought that makes me wonder: we have 117
> cws with more or less unfinished work. I doubt that we will integrate
> them all anytime soon, as integration also comprises developing the
> merged code further until it has sufficient quality for becoming part of
> the trunk. [Wat is "sufficient" is still undefined - we surely won't
> continue the overdone QA approval process from the "old" OOo project,
> but OTOH also shouldn't throw code at the repository at will.]
>
> Some of the 117 cws are anbandoned work, others are work in an early
> state that most probably doesn't make sense to be continued without the
> developers starting it.
>
> Do we really want to have code in the svn repo that will never be used?
> The alternative would be to add cws to svn only after review.
>
Right. That is why I was thinking that maybe we just create an
archival copy of the entire repository, including all CWS, and host
that as a read only Hg or git instance. Then migrate the trunk to
SVN, If there are some CWS that we know are already approved for
3.4, then include those as well.
That way, if someone does come by, months later, and decide they want
to complete work CWS, then they can still clone them and work on them.
But then they would need to copy them into a SVN working copy, and
merge and commit from there. Obviously, this does complicate things
for the future CWS developers. But they are in the best position to
stabilize and merge their work.
-Rob
> Regards,
> Mathias
>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Thorsten Behrens <th...@documentfoundation.org>.
Mathias Bauer wrote:
> Do we really want to have code in the svn repo that will never be used?
> The alternative would be to add cws to svn only after review.
>
A somewhat related question would be, until when will the Oracle
offer to extend the source code grant last? Since work done by
Oracle developers would otherwise be effectively unusable for
Apache, even *if* someone then later comes & picks it up?
Cheers,
-- Thorsten, who therefore sees some merit in reviewing them all
Re: fetch-all-cws.sh
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 07/05/2011 10:48 PM, schrieb Mathias Bauer:
> On 05.07.2011 22:20, Marcus (OOo) wrote:
>
>> Am 07/05/2011 09:33 PM, schrieb Mathias Bauer:
>>> On 05.07.2011 18:14, Mathias Bauer wrote:
>>>
>>>> It seems that my memory had fooled me: so if anybody can create an svn
>>>> dump file, I will try to recap what we have agreed to so far and have a
>>>> look into the conversion. In case anyone else is already at this, please
>>>> let me know.
>>> Having said that, there's a thought that makes me wonder: we have 117
>>> cws with more or less unfinished work. I doubt that we will integrate
>>> them all anytime soon, as integration also comprises developing the
>>> merged code further until it has sufficient quality for becoming part of
>>> the trunk. [Wat is "sufficient" is still undefined - we surely won't
>>> continue the overdone QA approval process from the "old" OOo project,
>>> but OTOH also shouldn't throw code at the repository at will.]
>>>
>>> Some of the 117 cws are anbandoned work, others are work in an early
>>> state that most probably doesn't make sense to be continued without the
>>> developers starting it.
>>>
>>> Do we really want to have code in the svn repo that will never be used?
>>> The alternative would be to add cws to svn only after review.
>>
>> IMHO you should say "Do we really want to have code in the svn repo that
>> will /maybe/ never be used?". Then I would say "yes, until it's really
>> clear that the work in the specific CWS is not continued resp. will be
>> finished.
>>
>> Maybe the developers will come to us at a later point in time? Or other
>> dev's would like to go on with the started work?
>>
>> I don't know but maybe it's relatively easy to investigate how far the
>> CWS is in its development? If just started then it could be really
>> dropped. But when a lot is already done, then we should keep it for x
>> months (x = tdbd).
>>
>> When we define to throw it away now (or better said we won't transfer to
>> the files from the Orcale server) then there is no chance at all for this.
>
> I don't opt for throwing anything away, my thoughts went into the same
> direction as Rob's. Keeping all cws in a read-only git repository until
> someone actually cares for working on one of them makes a lot of sense
> to me.
OK, after a second read I've now understood your last sentence ("The
alternative would be to add cws to svn only after review."). ;-)
> Of course there are some cws that will be worked on soon or that are
> already finished. We can move them to svn now, but others may follow later.
Great, then our thinking goes into the same direction.
Marcus
Re: fetch-all-cws.sh
Posted by Mathias Bauer <Ma...@gmx.net>.
On 05.07.2011 22:20, Marcus (OOo) wrote:
> Am 07/05/2011 09:33 PM, schrieb Mathias Bauer:
>> On 05.07.2011 18:14, Mathias Bauer wrote:
>>
>>> It seems that my memory had fooled me: so if anybody can create an svn
>>> dump file, I will try to recap what we have agreed to so far and have a
>>> look into the conversion. In case anyone else is already at this, please
>>> let me know.
>> Having said that, there's a thought that makes me wonder: we have 117
>> cws with more or less unfinished work. I doubt that we will integrate
>> them all anytime soon, as integration also comprises developing the
>> merged code further until it has sufficient quality for becoming part of
>> the trunk. [Wat is "sufficient" is still undefined - we surely won't
>> continue the overdone QA approval process from the "old" OOo project,
>> but OTOH also shouldn't throw code at the repository at will.]
>>
>> Some of the 117 cws are anbandoned work, others are work in an early
>> state that most probably doesn't make sense to be continued without the
>> developers starting it.
>>
>> Do we really want to have code in the svn repo that will never be used?
>> The alternative would be to add cws to svn only after review.
>
> IMHO you should say "Do we really want to have code in the svn repo that
> will /maybe/ never be used?". Then I would say "yes, until it's really
> clear that the work in the specific CWS is not continued resp. will be
> finished.
>
> Maybe the developers will come to us at a later point in time? Or other
> dev's would like to go on with the started work?
>
> I don't know but maybe it's relatively easy to investigate how far the
> CWS is in its development? If just started then it could be really
> dropped. But when a lot is already done, then we should keep it for x
> months (x = tdbd).
>
> When we define to throw it away now (or better said we won't transfer to
> the files from the Orcale server) then there is no chance at all for this.
I don't opt for throwing anything away, my thoughts went into the same
direction as Rob's. Keeping all cws in a read-only git repository until
someone actually cares for working on one of them makes a lot of sense
to me.
Of course there are some cws that will be worked on soon or that are
already finished. We can move them to svn now, but others may follow later.
Regards,
Mathias
Re: fetch-all-cws.sh
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 07/05/2011 09:33 PM, schrieb Mathias Bauer:
> On 05.07.2011 18:14, Mathias Bauer wrote:
>
>> It seems that my memory had fooled me: so if anybody can create an svn
>> dump file, I will try to recap what we have agreed to so far and have a
>> look into the conversion. In case anyone else is already at this, please
>> let me know.
> Having said that, there's a thought that makes me wonder: we have 117
> cws with more or less unfinished work. I doubt that we will integrate
> them all anytime soon, as integration also comprises developing the
> merged code further until it has sufficient quality for becoming part of
> the trunk. [Wat is "sufficient" is still undefined - we surely won't
> continue the overdone QA approval process from the "old" OOo project,
> but OTOH also shouldn't throw code at the repository at will.]
>
> Some of the 117 cws are anbandoned work, others are work in an early
> state that most probably doesn't make sense to be continued without the
> developers starting it.
>
> Do we really want to have code in the svn repo that will never be used?
> The alternative would be to add cws to svn only after review.
IMHO you should say "Do we really want to have code in the svn repo that
will /maybe/ never be used?". Then I would say "yes, until it's really
clear that the work in the specific CWS is not continued resp. will be
finished.
Maybe the developers will come to us at a later point in time? Or other
dev's would like to go on with the started work?
I don't know but maybe it's relatively easy to investigate how far the
CWS is in its development? If just started then it could be really
dropped. But when a lot is already done, then we should keep it for x
months (x = tdbd).
When we define to throw it away now (or better said we won't transfer to
the files from the Orcale server) then there is no chance at all for this.
My 2 ct from a non-developer point of view.
Marcus
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
Moin,
On 05.07.2011 18:14, Mathias Bauer wrote:
> It seems that my memory had fooled me: so if anybody can create an svn
> dump file, I will try to recap what we have agreed to so far and have a
> look into the conversion. In case anyone else is already at this, please
> let me know.
Having said that, there's a thought that makes me wonder: we have 117
cws with more or less unfinished work. I doubt that we will integrate
them all anytime soon, as integration also comprises developing the
merged code further until it has sufficient quality for becoming part of
the trunk. [Wat is "sufficient" is still undefined - we surely won't
continue the overdone QA approval process from the "old" OOo project,
but OTOH also shouldn't throw code at the repository at will.]
Some of the 117 cws are anbandoned work, others are work in an early
state that most probably doesn't make sense to be continued without the
developers starting it.
Do we really want to have code in the svn repo that will never be used?
The alternative would be to add cws to svn only after review.
Regards,
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
It seems that my memory had fooled me: so if anybody can create an svn
dump file, I will try to recap what we have agreed to so far and have a
look into the conversion. In case anyone else is already at this, please
let me know.
Regards,
Mathias
On 05.07.2011 17:58, Joe Schaefer wrote:
> Apache is all about empowerment of volunteers. If people
> step up, progress will happen. If people don't, it won't.
>
>
> Part of Apache style leadership is the ability to induce
> people to volunteer, as well as to create discussion that
> leads to consensus. Both types of activities are vital
> at this point in the life of the project, but relatively
> few people are participating in that mold.
>
> Your mentors are here to guide, not to direct. It is up
> to the people on this list to determine the fate of these
> issues.
>
> HTH
>
>
> ----- Original Message ----
>> From: Pedro F. Giffuni <gi...@tutopia.com>
>> To: ooo-dev@incubator.apache.org
>> Sent: Tue, July 5, 2011 11:53:14 AM
>> Subject: Re: fetch-all-cws.sh (was: Building a single Hg repository)
>>
>> I agree with Mathias ..
>>
>> We are not advancing simply because it's not clear who does what.
>>
>> - Greg made a script to import the Hg stuff ... who will run it?
>> - Matthias made a list, is this what Oracle should relicense, or
>> perhaps we should wait until we merge the resulting SVN branches.
>> - Who takes the list to Oracle.
>> - Bugzilla or JIRA?
>> - Who can provide the MediaWiki stuff so we actually translate
>> it to confluence?
>>
>> We really need people taking the tasks at hand.
>>
>> cheers,
>>
>> Pedro.
>>
>> --- On Tue, 7/5/11, Joe Schaefer <jo...@yahoo.com> wrote:
>>
>> > From: Joe Schaefer <jo...@yahoo.com>
>> > Subject: Re: fetch-all-cws.sh (was: Building a single Hg repository)
>> > To: ooo-dev@incubator.apache.org
>> > Date: Tuesday, July 5, 2011, 9:58 AM
>> >
>> >
>> > ----- Original Message ----
>> > > From: Mathias Bauer <Ma...@gmx.net>
>> > > To: ooo-dev@incubator.apache.org
>> > > Sent: Tue, July 5, 2011 10:55:25 AM
>> > > Subject: Re: fetch-all-cws.sh (was: Building a single
>> > Hg repository)
>> > >
>> > > On 01.07.2011 23:18, Greg Stein wrote:
>> > >
>> > > > Right. I think we just bring it all over,
>> > and then sort it out in our
>> > > > repository.
>> > > May I ask who will be the one that "brings it
>> > all over"? I don't find
>> > > the reference, but from past reading I
>> > understood that this is nothing
>> > > that one of us "regular committers" can do.
>> >
>> > Why not? It's DVCS after all, so long as we have a
>> > *repeatable* process
>> > for populating an svn dump file, and there's a way to map
>> > that process over
>> > to whatever file list Oracle is willing to provide, seems
>> > like any developer
>> > can initiate this process.
>> >
>> >
>>
>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Joe Schaefer <jo...@yahoo.com>.
Apache is all about empowerment of volunteers. If people
step up, progress will happen. If people don't, it won't.
Part of Apache style leadership is the ability to induce
people to volunteer, as well as to create discussion that
leads to consensus. Both types of activities are vital
at this point in the life of the project, but relatively
few people are participating in that mold.
Your mentors are here to guide, not to direct. It is up
to the people on this list to determine the fate of these
issues.
HTH
----- Original Message ----
> From: Pedro F. Giffuni <gi...@tutopia.com>
> To: ooo-dev@incubator.apache.org
> Sent: Tue, July 5, 2011 11:53:14 AM
> Subject: Re: fetch-all-cws.sh (was: Building a single Hg repository)
>
> I agree with Mathias ..
>
> We are not advancing simply because it's not clear who does what.
>
> - Greg made a script to import the Hg stuff ... who will run it?
> - Matthias made a list, is this what Oracle should relicense, or
> perhaps we should wait until we merge the resulting SVN branches.
> - Who takes the list to Oracle.
> - Bugzilla or JIRA?
> - Who can provide the MediaWiki stuff so we actually translate
> it to confluence?
>
> We really need people taking the tasks at hand.
>
> cheers,
>
> Pedro.
>
> --- On Tue, 7/5/11, Joe Schaefer <jo...@yahoo.com> wrote:
>
> > From: Joe Schaefer <jo...@yahoo.com>
> > Subject: Re: fetch-all-cws.sh (was: Building a single Hg repository)
> > To: ooo-dev@incubator.apache.org
> > Date: Tuesday, July 5, 2011, 9:58 AM
> >
> >
> > ----- Original Message ----
> > > From: Mathias Bauer <Ma...@gmx.net>
> > > To: ooo-dev@incubator.apache.org
> > > Sent: Tue, July 5, 2011 10:55:25 AM
> > > Subject: Re: fetch-all-cws.sh (was: Building a single
> > Hg repository)
> > >
> > > On 01.07.2011 23:18, Greg Stein wrote:
> > >
> > > > Right. I think we just bring it all over,
> > and then sort it out in our
> > > > repository.
> > > May I ask who will be the one that "brings it
> > all over"? I don't find
> > > the reference, but from past reading I
> > understood that this is nothing
> > > that one of us "regular committers" can do.
> >
> > Why not? It's DVCS after all, so long as we have a
> > *repeatable* process
> > for populating an svn dump file, and there's a way to map
> > that process over
> > to whatever file list Oracle is willing to provide, seems
> > like any developer
> > can initiate this process.
> >
> >
>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Dave Fisher <da...@comcast.net>.
Hi Andrew,
Thanks for the update.
> On 7/5/2011 8:53 AM, Pedro F. Giffuni wrote:
>> I agree with Mathias ..
>>
>> We are not advancing simply because it's not clear who does what.
>>
>> - Greg made a script to import the Hg stuff ... who will run it?
> A number of us are - and props gentlemen, the scripts work great.
>
>> - Matthias made a list, is this what Oracle should relicense, or
>> perhaps we should wait until we merge the resulting SVN branches.
>> - Who takes the list to Oracle.
> As discussed before on the list, I am tracking all of the resources that have been identified and will be working this all through on the Oracle side.
> The list goes beyond just code, to artwork, bz database, translations, the forums, wikis, and more.
If possible would you provide some more detail in the wiki - probably on one of these pages - https://cwiki.apache.org/confluence/label/OOOUSERS/site
About artwork, I found these pages the other day:
http://www.openoffice.org/trademark/brandrefresh.html
http://ui.openoffice.org/VisualDesign/OOo3_refresh.html
http://patentpending.co.nz/2010/04/04/openoffice-org-new-setup-file-and-application-icons/
I was wondering when we might get our hands on the original photoshop (or is it something else?) files for the logos.
>> - Bugzilla or JIRA?
> (I personally lean JIRA)
Since the BZ is behind and has a non-standard schema, maybe the next step is to understand the actual altered schema vs. both the JIRA and BZ 4.0 schemas. Once we know that the work to convert can be determined. Mark Thomas had some ideas, he's the guy on the Apache Infra for BZ/JIRA.
>> - Who can provide the MediaWiki stuff so we actually translate
>> it to confluence?
> working on this, also.
Do you think that the UWC[1] will be useful? Or, will we need to make up our own translation?
Regards,
Dave
[1] https://studio.plugins.atlassian.com/wiki/display/UWC/UWC+User+Documentation
>>
>> We really need people taking the tasks at hand.
>>
>> cheers,
>>
>> Pedro.
>>
>> --- On Tue, 7/5/11, Joe Schaefer<jo...@yahoo.com> wrote:
>>
>>> From: Joe Schaefer<jo...@yahoo.com>
>>> Subject: Re: fetch-all-cws.sh (was: Building a single Hg repository)
>>> To: ooo-dev@incubator.apache.org
>>> Date: Tuesday, July 5, 2011, 9:58 AM
>>>
>>>
>>> ----- Original Message ----
>>>> From: Mathias Bauer<Ma...@gmx.net>
>>>> To: ooo-dev@incubator.apache.org
>>>> Sent: Tue, July 5, 2011 10:55:25 AM
>>>> Subject: Re: fetch-all-cws.sh (was: Building a single
>>> Hg repository)
>>>> On 01.07.2011 23:18, Greg Stein wrote:
>>>>
>>>>> Right. I think we just bring it all over,
>>> and then sort it out in our
>>>>> repository.
>>>> May I ask who will be the one that "brings it
>>> all over"? I don't find
>>>> the reference, but from past reading I
>>> understood that this is nothing
>>>> that one of us "regular committers" can do.
>>> Why not? It's DVCS after all, so long as we have a
>>> *repeatable* process
>>> for populating an svn dump file, and there's a way to map
>>> that process over
>>> to whatever file list Oracle is willing to provide, seems
>>> like any developer
>>> can initiate this process.
>>>
>>>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by "Pedro F. Giffuni" <gi...@tutopia.com>.
Andrew;
Thank you and Raphael for the update ...
I was just clueless about someone was doing the real work!
I have been an OOo user for a while but only now some of us
are starting to get the grasp of the *HUGE* contribution, not
only in software but in resources, that SUN/Oracle has made
during these years to free software.
Pedro.
--- On Wed, 7/6/11, Andrew Rist wrote:
>
> On 7/5/2011 8:53 AM, Pedro F. Giffuni wrote:
> > I agree with Mathias ..
> >
> > We are not advancing simply because it's not clear who
> does what.
> >
> > - Greg made a script to import the Hg stuff ... who
> will run it?
> A number of us are - and props gentlemen, the scripts work
> great.
>
> > - Matthias made a list, is this what Oracle should
> relicense, or
> > perhaps we should wait until we merge the resulting
> SVN branches.
> > - Who takes the list to Oracle.
> As discussed before on the list, I am tracking all of the
> resources that
> have been identified and will be working this all through
> on the Oracle
> side.
> The list goes beyond just code, to artwork, bz database,
> translations,
> the forums, wikis, and more.
>
> > - Bugzilla or JIRA?
> (I personally lean JIRA)
> > - Who can provide the MediaWiki stuff so we actually
> translate
> > it to confluence?
> working on this, also.
> >
> > We really need people taking the tasks at hand.
> >
> > cheers,
> >
> > Pedro.
>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Andrew Rist <an...@oracle.com>.
On 7/5/2011 8:53 AM, Pedro F. Giffuni wrote:
> I agree with Mathias ..
>
> We are not advancing simply because it's not clear who does what.
>
> - Greg made a script to import the Hg stuff ... who will run it?
A number of us are - and props gentlemen, the scripts work great.
> - Matthias made a list, is this what Oracle should relicense, or
> perhaps we should wait until we merge the resulting SVN branches.
> - Who takes the list to Oracle.
As discussed before on the list, I am tracking all of the resources that
have been identified and will be working this all through on the Oracle
side.
The list goes beyond just code, to artwork, bz database, translations,
the forums, wikis, and more.
> - Bugzilla or JIRA?
(I personally lean JIRA)
> - Who can provide the MediaWiki stuff so we actually translate
> it to confluence?
working on this, also.
>
> We really need people taking the tasks at hand.
>
> cheers,
>
> Pedro.
>
> --- On Tue, 7/5/11, Joe Schaefer<jo...@yahoo.com> wrote:
>
>> From: Joe Schaefer<jo...@yahoo.com>
>> Subject: Re: fetch-all-cws.sh (was: Building a single Hg repository)
>> To: ooo-dev@incubator.apache.org
>> Date: Tuesday, July 5, 2011, 9:58 AM
>>
>>
>> ----- Original Message ----
>>> From: Mathias Bauer<Ma...@gmx.net>
>>> To: ooo-dev@incubator.apache.org
>>> Sent: Tue, July 5, 2011 10:55:25 AM
>>> Subject: Re: fetch-all-cws.sh (was: Building a single
>> Hg repository)
>>> On 01.07.2011 23:18, Greg Stein wrote:
>>>
>>>> Right. I think we just bring it all over,
>> and then sort it out in our
>>>> repository.
>>> May I ask who will be the one that "brings it
>> all over"? I don't find
>>> the reference, but from past reading I
>> understood that this is nothing
>>> that one of us "regular committers" can do.
>> Why not? It's DVCS after all, so long as we have a
>> *repeatable* process
>> for populating an svn dump file, and there's a way to map
>> that process over
>> to whatever file list Oracle is willing to provide, seems
>> like any developer
>> can initiate this process.
>>
>>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by "Pedro F. Giffuni" <gi...@tutopia.com>.
I agree with Mathias ..
We are not advancing simply because it's not clear who does what.
- Greg made a script to import the Hg stuff ... who will run it?
- Matthias made a list, is this what Oracle should relicense, or
perhaps we should wait until we merge the resulting SVN branches.
- Who takes the list to Oracle.
- Bugzilla or JIRA?
- Who can provide the MediaWiki stuff so we actually translate
it to confluence?
We really need people taking the tasks at hand.
cheers,
Pedro.
--- On Tue, 7/5/11, Joe Schaefer <jo...@yahoo.com> wrote:
> From: Joe Schaefer <jo...@yahoo.com>
> Subject: Re: fetch-all-cws.sh (was: Building a single Hg repository)
> To: ooo-dev@incubator.apache.org
> Date: Tuesday, July 5, 2011, 9:58 AM
>
>
> ----- Original Message ----
> > From: Mathias Bauer <Ma...@gmx.net>
> > To: ooo-dev@incubator.apache.org
> > Sent: Tue, July 5, 2011 10:55:25 AM
> > Subject: Re: fetch-all-cws.sh (was: Building a single
> Hg repository)
> >
> > On 01.07.2011 23:18, Greg Stein wrote:
> >
> > > Right. I think we just bring it all over,
> and then sort it out in our
> > > repository.
> > May I ask who will be the one that "brings it
> all over"? I don't find
> > the reference, but from past reading I
> understood that this is nothing
> > that one of us "regular committers" can do.
>
> Why not? It's DVCS after all, so long as we have a
> *repeatable* process
> for populating an svn dump file, and there's a way to map
> that process over
> to whatever file list Oracle is willing to provide, seems
> like any developer
> can initiate this process.
>
>
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Joe Schaefer <jo...@yahoo.com>.
----- Original Message ----
> From: Mathias Bauer <Ma...@gmx.net>
> To: ooo-dev@incubator.apache.org
> Sent: Tue, July 5, 2011 10:55:25 AM
> Subject: Re: fetch-all-cws.sh (was: Building a single Hg repository)
>
> On 01.07.2011 23:18, Greg Stein wrote:
>
> > Right. I think we just bring it all over, and then sort it out in our
> > repository.
> May I ask who will be the one that "brings it all over"? I don't find
> the reference, but from past reading I understood that this is nothing
> that one of us "regular committers" can do.
Why not? It's DVCS after all, so long as we have a *repeatable* process
for populating an svn dump file, and there's a way to map that process over
to whatever file list Oracle is willing to provide, seems like any developer
can initiate this process.
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Mathias Bauer <Ma...@gmx.net>.
On 01.07.2011 23:18, Greg Stein wrote:
> Right. I think we just bring it all over, and then sort it out in our
> repository.
May I ask who will be the one that "brings it all over"? I don't find
the reference, but from past reading I understood that this is nothing
that one of us "regular committers" can do.
As it seems, we have "lazy consensus" about the approach to fetch OOO340
master and all cws and then convert to svn. The list of cws to fetch has
been reviewed by at least two developers that are familiar with the old
code base, so for me this list is complete.
When will the import happen? Is there anything where I can help? I
really want to get some code. :-)
Regards,
Mathias
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Michael Stahl <ms...@openoffice.org>.
On 02.07.2011 04:14, Greg Stein wrote:
> On Fri, Jul 1, 2011 at 17:18, Greg Stein<gs...@gmail.com> wrote:
>> On Fri, Jul 1, 2011 at 16:58, Michael Stahl<ms...@openoffice.org> wrote:
>> ...
>>> but actually i think that a lot of these 250 CWSes will not contain a
>>> changeset that is not in the master already; a lot of developers create new
>>> CWS and then (have to) work on something else for some weeks...
>>>
>>> so i have adapted the fetch script to skip empty CWSes.
>>
>> Saw that. Great!
>
> I used that code to figure out which are empty, and then commented
> those out from the cws-list.txt file. I think it is best to keep the
> code in there, regardless. Somebody may want run it in the future.
>
> When I was working on this, however, I noticed we were not attempting
> to fetch the cws_l10n repositories. So I fixed the filter, but
> Mercurial said those repositories are not related to DEV300. What is
> going on there, and what do we need to fix? Should we be fetching the
> cws_l10n/* repositories? And if so... how do we integrate that into
> our single repository, and then into our svn repository?
these need to be pulled into the l10n repository, not the main repository:
http://hg.services.openoffice.org/master_l10n/OOO340/
and i'd expect almost all of these to be empty.
the l10n repo contains a single top-level module.
we could integrate these in SVN in 2 ways:
1. like we did in the old times, as a top-level module next to all
the other ones:
ooo/trunk/{hundreds of modules}
ooo/trunk/l10n
2. under a separate root:
ooo/trunk/{hundreds of modules}
ooo-l10n/trunk/l10n
1. has the advantage that everything can be branched/tagged in one
operation, while 2. has the advantage that it's easier for developers
that only build en-US to not waste disk space/build time (the build
system is already prepared to have it "optional").
(of course there are other ways we could split up the modules, like e.g.
have a separate Apache URE product upon which Apache OOo can depend, but
that's not something we should discuss now...)
--
"Inexact sciences like economics advance funeral by funeral."
-- Paul Samuelson
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Greg Stein <gs...@gmail.com>.
On Fri, Jul 1, 2011 at 17:18, Greg Stein <gs...@gmail.com> wrote:
> On Fri, Jul 1, 2011 at 16:58, Michael Stahl <ms...@openoffice.org> wrote:
>...
>> but actually i think that a lot of these 250 CWSes will not contain a
>> changeset that is not in the master already; a lot of developers create new
>> CWS and then (have to) work on something else for some weeks...
>>
>> so i have adapted the fetch script to skip empty CWSes.
>
> Saw that. Great!
I used that code to figure out which are empty, and then commented
those out from the cws-list.txt file. I think it is best to keep the
code in there, regardless. Somebody may want run it in the future.
When I was working on this, however, I noticed we were not attempting
to fetch the cws_l10n repositories. So I fixed the filter, but
Mercurial said those repositories are not related to DEV300. What is
going on there, and what do we need to fix? Should we be fetching the
cws_l10n/* repositories? And if so... how do we integrate that into
our single repository, and then into our svn repository?
Cheers,
-g
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Greg Stein <gs...@gmail.com>.
On Fri, Jul 1, 2011 at 16:58, Michael Stahl <ms...@openoffice.org> wrote:
> On 01.07.2011 13:42, Greg Stein wrote:
>...
>> This is the approach that I took. Please look at
>> tools/dev/fetch-all-cws.sh. Each of these CWS repositories (on Mac OS)
>> are consuming 600 Mb *minimum*. I've fetched a dozen, and a couple are
>> over 2 Gb each, and another over 1 Gb. And this is with the clone/pull
>> technique.
>
> indeed, i get similar numbers.
> a clone with hardlinks is 34 MB on my filesystem.
> a CWS with a single changeset takes 670MB.
> reason is that for every commit, 2 files that store changelogs and manifests
> are modified, and together these are >600 MB in our repo.
While away from the laptop, I realized that I may have messed up the
counting due to the hardlinks. But I find that 'du' counts files just
once per execution, which is the right thing.
So it is good to see you have an explanation for why it takes so much space.
>...
> but actually i think that a lot of these 250 CWSes will not contain a
> changeset that is not in the master already; a lot of developers create new
> CWS and then (have to) work on something else for some weeks...
>
> so i have adapted the fetch script to skip empty CWSes.
Saw that. Great!
>
>> A possible alternative to pulling N repositories, then combining them
>> in a second step, is to attempt to bring them all into a single
>> repository, one at a time. This is a little more scary for me, not
>> knowing Hg, to understand how restartable and repeatable this process
>> will be in the face of errors. Either starting from scratch, or (I
>> believe an important feature) if it needs to be resumed after some
>> minor failure (eg. network failure).
>
> this would of course take much less space, but then it would be necessary to
> mark the newly pulled head immediately to know which CWS it corresponds to.
Could you review Herbert's script? I think we'd actually put that into
single-hg.sh, and delete the fetch-all-cws.sh script as unneeded.
Is that a safe approach? What happens if it stops part way through?
Can it be restarted with minimal effort? And what happens to the
bookmarks in such a case?
>> We have a script. It is time to make it work.
>>
>> Michael: you say that some CWS repositories are useless. If so, then
>> please update tools/dev/cws-list.txt to comment-out those CWS's with
>> some explanation for future readers. No need for us to attempt to
>> process them if they're bogus.
>
> i have checked the status in EIS, and it seems like the repos for almost all
> integrated/deleted CWSes have already been automatically removed from the
> server.
> found a couple that were in a state "cancelled", which i didn't even know
> existed, sounds like we don't need those, so i've commented them out.
Great.
> of course some CWSes contain stuff that's not useful, but i don't know which
> these are :)
Right. I think we just bring it all over, and then sort it out in our
repository.
Cheers,
-g
Re: fetch-all-cws.sh (was: Building a single Hg repository)
Posted by Michael Stahl <ms...@openoffice.org>.
On 01.07.2011 13:42, Greg Stein wrote:
> On Wed, Jun 29, 2011 at 05:04, Michael Stahl<ms...@openoffice.org> wrote:
>> ...
>> in principle the size of a CWS is on the same order as the master, because
>> it's just another HG repository.
>>
>> but HG supports hardlinks between repositories (in newer versions even on
>> win32), so you can "hg clone" the master on the same filesystem and then
>> pull in the CWS, and it will be _much_ faster and take _much_ less
>> additional space
>
> This is the approach that I took. Please look at
> tools/dev/fetch-all-cws.sh. Each of these CWS repositories (on Mac OS)
> are consuming 600 Mb *minimum*. I've fetched a dozen, and a couple are
> over 2 Gb each, and another over 1 Gb. And this is with the clone/pull
> technique.
indeed, i get similar numbers.
a clone with hardlinks is 34 MB on my filesystem.
a CWS with a single changeset takes 670MB.
reason is that for every commit, 2 files that store changelogs and
manifests are modified, and together these are >600 MB in our repo.
> I don't have enough space on my laptop to do a complete trial run. I'm
> hoping that somebody can figure out how to reduce the disk footprint,
> or determine that we just have to suck it up. And it would be nice to
> understand what that target size will be, for all 250 CWS
> repositories.
i think i wrote that all CWSes as HG repos take ~100 GB, but actually i
now think i remembered wrong and the number was more like ~150 GB.
(i did this originally in 2 steps, and i remembered only the second step...)
(and if it weren't so late now i'd even dig out my external hd and run
du...)
of course the filesystem used could make a difference here.
but actually i think that a lot of these 250 CWSes will not contain a
changeset that is not in the master already; a lot of developers create
new CWS and then (have to) work on something else for some weeks...
so i have adapted the fetch script to skip empty CWSes.
> A possible alternative to pulling N repositories, then combining them
> in a second step, is to attempt to bring them all into a single
> repository, one at a time. This is a little more scary for me, not
> knowing Hg, to understand how restartable and repeatable this process
> will be in the face of errors. Either starting from scratch, or (I
> believe an important feature) if it needs to be resumed after some
> minor failure (eg. network failure).
this would of course take much less space, but then it would be
necessary to mark the newly pulled head immediately to know which CWS it
corresponds to.
> We have a script. It is time to make it work.
>
> Michael: you say that some CWS repositories are useless. If so, then
> please update tools/dev/cws-list.txt to comment-out those CWS's with
> some explanation for future readers. No need for us to attempt to
> process them if they're bogus.
i have checked the status in EIS, and it seems like the repos for almost
all integrated/deleted CWSes have already been automatically removed
from the server.
found a couple that were in a state "cancelled", which i didn't even
know existed, sounds like we don't need those, so i've commented them out.
of course some CWSes contain stuff that's not useful, but i don't know
which these are :)
--
"Fools ignore complexity. Pragmatists suffer it. Some can avoid it.
Geniuses remove it." -- Alan J. Perlis