You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bloodhound.apache.org by Gary Martin <ga...@wandisco.com> on 2012/08/07 20:29:39 UTC
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Not sure if this could be of interest beyond Bloodhound. It is a very
simple script (and quite possibly over-complicated in reality).
The attachment referred to is here:
https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
Just tested it on 1229640 empty commits:
python3 emptyrevs.py 1229640 | eatmydata svnadmin
--bypass-prop-validation load repos/
which took about 2 hours - I suspect that the svnadmin load was the real
bottleneck.. it is just a few seconds to create about 92M of data if you
direct to a file instead.
Cheers,
Gary
On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
> #156: Local copy of bloodhound part of Apache repo for browse functionality
> ------------------------+-----------------------
> Reporter: gjm | Owner: nobody
> Type: task | Status: new
> Priority: critical | Milestone: Release 2
> Component: siteadmin | Version:
> Resolution: | Keywords:
> ------------------------+-----------------------
>
> Comment (by gjm):
>
> As part of my investigation around creating a mirror of the bloodhound
> portion, I have written a very simple script (only complicated when I
> decided to have a quick look at the argparse module and python2/3 issues),
> inspired by a suggestion from Philip Martin.
>
> I have attached that script [attachment:emptyrevs.py here]. It may be
> worth putting in the bloodhound repository in case we need it again of
> course.
>
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
On Aug 9, 2012 7:17 PM, "Gary Martin" <ga...@wandisco.com> wrote:
>
> On 09/08/12 16:36, Greg Stein wrote:
>>
>> On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>>>
>>> That sounds great. AFAIK an NFS mount will be fine for Allura to read
>>
>> from.
>>>
>>> By the way, I've subscribed to the bloodhound-dev list also. It seems
>>> that allura and bloodhound have a lot in common: python code/project
>>> management tools both going through incubation :)
>>
>> Hehe... actually, I've considered raising the idea of plugging BH into
>> Allura. Guess this is a good time to do just that :-)
>>
>> Cheers,
>> -g
>>
>
> Certainly a very interesting idea. Would this be considered consistent
with Allura's plans though?
No idea :-)
Somebody had to bring it up, though ;-) ... I imagine it would be an idea
for post-graduation, and when/if somebody gets the itch.
Cheers,
-g
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
On 10/08/12 21:05, Olemis Lang wrote:
>>>> Hehe... actually, I've considered raising the idea of plugging BH into
>>>> >>>Allura. Guess this is a good time to do just that:-)
>>>> >>>
>>>> >>>Cheers,
>>>> >>>-g
>>>> >>>
>>> >>
>>> >>Certainly a very interesting idea. Would this be considered consistent
>>> >>with Allura's plans though?
>>> >>
>> >
>> >A key part of Allura's architecture is the pluggability of "tools". A
>> >standalone BloodHound tool for Allura could be developed without
>> >affecting the Allura platform at all.
>> >
>> >That said, a lot of the benefit of Allura is in using the core models
>> >which would give you unified searching, permissions, cross-linking
>> >between tools, etc. So you couldn't reap those benefits without a lot
>> >of customization to BloodHound.
>> >
>> >
> AFAICS something in our schedule related to this may be #142 [1]_ , isn't it ?
>
> .. [1] #142 Product-specific permissions
> (https://issues.apache.org/bloodhound/ticket/142)
I don't think that is strictly related as that is about internal
permissions.
Although it will take a bit of investigation, I don't see a particular
problem with Bloodhound providing a plugin to tap into permissions,
another search mechanism and so on. I am not sure how long it will be
before we can turn our attention to this at the moment, but it is very
good to know that interoperability is a possibility.
Cheers,
Gary
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
On 10/08/12 21:05, Olemis Lang wrote:
>>>> Hehe... actually, I've considered raising the idea of plugging BH into
>>>> >>>Allura. Guess this is a good time to do just that:-)
>>>> >>>
>>>> >>>Cheers,
>>>> >>>-g
>>>> >>>
>>> >>
>>> >>Certainly a very interesting idea. Would this be considered consistent
>>> >>with Allura's plans though?
>>> >>
>> >
>> >A key part of Allura's architecture is the pluggability of "tools". A
>> >standalone BloodHound tool for Allura could be developed without
>> >affecting the Allura platform at all.
>> >
>> >That said, a lot of the benefit of Allura is in using the core models
>> >which would give you unified searching, permissions, cross-linking
>> >between tools, etc. So you couldn't reap those benefits without a lot
>> >of customization to BloodHound.
>> >
>> >
> AFAICS something in our schedule related to this may be #142 [1]_ , isn't it ?
>
> .. [1] #142 Product-specific permissions
> (https://issues.apache.org/bloodhound/ticket/142)
I don't think that is strictly related as that is about internal
permissions.
Although it will take a bit of investigation, I don't see a particular
problem with Bloodhound providing a plugin to tap into permissions,
another search mechanism and so on. I am not sure how long it will be
before we can turn our attention to this at the moment, but it is very
good to know that interoperability is a possibility.
Cheers,
Gary
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Olemis Lang <ol...@gmail.com>.
On 8/10/12, Dave Brondsema <da...@brondsema.net> wrote:
> On 8/9/12 7:17 PM, Gary Martin wrote:
>> On 09/08/12 16:36, Greg Stein wrote:
>>> On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>>>> That sounds great. AFAIK an NFS mount will be fine for Allura to read
>>> from.
>>>> By the way, I've subscribed to the bloodhound-dev list also. It seems
>>>> that allura and bloodhound have a lot in common: python code/project
>>>> management tools both going through incubation :)
>>> Hehe... actually, I've considered raising the idea of plugging BH into
>>> Allura. Guess this is a good time to do just that :-)
>>>
>>> Cheers,
>>> -g
>>>
>>
>> Certainly a very interesting idea. Would this be considered consistent
>> with Allura's plans though?
>>
>
> A key part of Allura's architecture is the pluggability of "tools". A
> standalone BloodHound tool for Allura could be developed without
> affecting the Allura platform at all.
>
> That said, a lot of the benefit of Allura is in using the core models
> which would give you unified searching, permissions, cross-linking
> between tools, etc. So you couldn't reap those benefits without a lot
> of customization to BloodHound.
>
>
AFAICS something in our schedule related to this may be #142 [1]_ , isn't it ?
.. [1] #142 Product-specific permissions
(https://issues.apache.org/bloodhound/ticket/142)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Olemis Lang <ol...@gmail.com>.
On 8/10/12, Dave Brondsema <da...@brondsema.net> wrote:
> On 8/9/12 7:17 PM, Gary Martin wrote:
>> On 09/08/12 16:36, Greg Stein wrote:
>>> On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>>>> That sounds great. AFAIK an NFS mount will be fine for Allura to read
>>> from.
>>>> By the way, I've subscribed to the bloodhound-dev list also. It seems
>>>> that allura and bloodhound have a lot in common: python code/project
>>>> management tools both going through incubation :)
>>> Hehe... actually, I've considered raising the idea of plugging BH into
>>> Allura. Guess this is a good time to do just that :-)
>>>
>>> Cheers,
>>> -g
>>>
>>
>> Certainly a very interesting idea. Would this be considered consistent
>> with Allura's plans though?
>>
>
> A key part of Allura's architecture is the pluggability of "tools". A
> standalone BloodHound tool for Allura could be developed without
> affecting the Allura platform at all.
>
> That said, a lot of the benefit of Allura is in using the core models
> which would give you unified searching, permissions, cross-linking
> between tools, etc. So you couldn't reap those benefits without a lot
> of customization to BloodHound.
>
>
AFAICS something in our schedule related to this may be #142 [1]_ , isn't it ?
.. [1] #142 Product-specific permissions
(https://issues.apache.org/bloodhound/ticket/142)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Dave Brondsema <da...@brondsema.net>.
On 8/9/12 7:17 PM, Gary Martin wrote:
> On 09/08/12 16:36, Greg Stein wrote:
>> On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>>> That sounds great. AFAIK an NFS mount will be fine for Allura to read
>> from.
>>> By the way, I've subscribed to the bloodhound-dev list also. It seems
>>> that allura and bloodhound have a lot in common: python code/project
>>> management tools both going through incubation :)
>> Hehe... actually, I've considered raising the idea of plugging BH into
>> Allura. Guess this is a good time to do just that :-)
>>
>> Cheers,
>> -g
>>
>
> Certainly a very interesting idea. Would this be considered consistent
> with Allura's plans though?
>
A key part of Allura's architecture is the pluggability of "tools". A
standalone BloodHound tool for Allura could be developed without
affecting the Allura platform at all.
That said, a lot of the benefit of Allura is in using the core models
which would give you unified searching, permissions, cross-linking
between tools, etc. So you couldn't reap those benefits without a lot
of customization to BloodHound.
--
Dave Brondsema : dave@brondsema.net
http://www.brondsema.net : personal
http://www.splike.com : programming
<><
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Dave Brondsema <da...@brondsema.net>.
On 8/9/12 7:17 PM, Gary Martin wrote:
> On 09/08/12 16:36, Greg Stein wrote:
>> On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>>> That sounds great. AFAIK an NFS mount will be fine for Allura to read
>> from.
>>> By the way, I've subscribed to the bloodhound-dev list also. It seems
>>> that allura and bloodhound have a lot in common: python code/project
>>> management tools both going through incubation :)
>> Hehe... actually, I've considered raising the idea of plugging BH into
>> Allura. Guess this is a good time to do just that :-)
>>
>> Cheers,
>> -g
>>
>
> Certainly a very interesting idea. Would this be considered consistent
> with Allura's plans though?
>
A key part of Allura's architecture is the pluggability of "tools". A
standalone BloodHound tool for Allura could be developed without
affecting the Allura platform at all.
That said, a lot of the benefit of Allura is in using the core models
which would give you unified searching, permissions, cross-linking
between tools, etc. So you couldn't reap those benefits without a lot
of customization to BloodHound.
--
Dave Brondsema : dave@brondsema.net
http://www.brondsema.net : personal
http://www.splike.com : programming
<><
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
On Aug 9, 2012 7:17 PM, "Gary Martin" <ga...@wandisco.com> wrote:
>
> On 09/08/12 16:36, Greg Stein wrote:
>>
>> On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>>>
>>> That sounds great. AFAIK an NFS mount will be fine for Allura to read
>>
>> from.
>>>
>>> By the way, I've subscribed to the bloodhound-dev list also. It seems
>>> that allura and bloodhound have a lot in common: python code/project
>>> management tools both going through incubation :)
>>
>> Hehe... actually, I've considered raising the idea of plugging BH into
>> Allura. Guess this is a good time to do just that :-)
>>
>> Cheers,
>> -g
>>
>
> Certainly a very interesting idea. Would this be considered consistent
with Allura's plans though?
No idea :-)
Somebody had to bring it up, though ;-) ... I imagine it would be an idea
for post-graduation, and when/if somebody gets the itch.
Cheers,
-g
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
On 09/08/12 16:36, Greg Stein wrote:
> On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>> That sounds great. AFAIK an NFS mount will be fine for Allura to read
> from.
>> By the way, I've subscribed to the bloodhound-dev list also. It seems
>> that allura and bloodhound have a lot in common: python code/project
>> management tools both going through incubation :)
> Hehe... actually, I've considered raising the idea of plugging BH into
> Allura. Guess this is a good time to do just that :-)
>
> Cheers,
> -g
>
Certainly a very interesting idea. Would this be considered consistent
with Allura's plans though?
Cheers,
Gary
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
On 09/08/12 16:36, Greg Stein wrote:
> On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>> That sounds great. AFAIK an NFS mount will be fine for Allura to read
> from.
>> By the way, I've subscribed to the bloodhound-dev list also. It seems
>> that allura and bloodhound have a lot in common: python code/project
>> management tools both going through incubation :)
> Hehe... actually, I've considered raising the idea of plugging BH into
> Allura. Guess this is a good time to do just that :-)
>
> Cheers,
> -g
>
Certainly a very interesting idea. Would this be considered consistent
with Allura's plans though?
Cheers,
Gary
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>
> That sounds great. AFAIK an NFS mount will be fine for Allura to read
from.
>
> By the way, I've subscribed to the bloodhound-dev list also. It seems
> that allura and bloodhound have a lot in common: python code/project
> management tools both going through incubation :)
Hehe... actually, I've considered raising the idea of plugging BH into
Allura. Guess this is a good time to do just that :-)
Cheers,
-g
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
On Aug 9, 2012 11:33 AM, "Dave Brondsema" <da...@brondsema.net> wrote:
>
> That sounds great. AFAIK an NFS mount will be fine for Allura to read
from.
>
> By the way, I've subscribed to the bloodhound-dev list also. It seems
> that allura and bloodhound have a lot in common: python code/project
> management tools both going through incubation :)
Hehe... actually, I've considered raising the idea of plugging BH into
Allura. Guess this is a good time to do just that :-)
Cheers,
-g
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Dave Brondsema <da...@brondsema.net>.
That sounds great. AFAIK an NFS mount will be fine for Allura to read from.
By the way, I've subscribed to the bloodhound-dev list also. It seems
that allura and bloodhound have a lot in common: python code/project
management tools both going through incubation :)
-Dave
On 8/8/12 5:26 PM, Greg Stein wrote:
> Very cool!
>
> But with that said... I was on IRC, and the Infra guys might actually
> create a full repository mirror for us. The thing is that Apache
> Allura (also incubating) will want a copy of their project(s), too.
> Tho... I think they're going with Git, but the concept is the same.
> Bloodhound and Allura both need local read-access to repositories.
>
> Infra was thinking about sync'ing repository copies over to a box (I
> forget the name). The BH and Allura VMs would be migrated over to that
> same box. The repositories would then be exposed via NFS, and the two
> VMs would (locally) mount that NFS share.
>
> I believe the Right Answer here is for both projects to confirm that
> this option is workable, and for us to ask Infra to start setting it
> up.
>
> Thoughts?
>
> Cheers,
> -g
>
> On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin <ga...@wandisco.com> wrote:
>> Not sure if this could be of interest beyond Bloodhound. It is a very simple
>> script (and quite possibly over-complicated in reality).
>>
>> The attachment referred to is here:
>> https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
>>
>> Just tested it on 1229640 empty commits:
>>
>> python3 emptyrevs.py 1229640 | eatmydata svnadmin
>> --bypass-prop-validation load repos/
>>
>> which took about 2 hours - I suspect that the svnadmin load was the real
>> bottleneck.. it is just a few seconds to create about 92M of data if you
>> direct to a file instead.
>>
>> Cheers,
>> Gary
>>
>>
>>
>> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>>>
>>> #156: Local copy of bloodhound part of Apache repo for browse
>>> functionality
>>> ------------------------+-----------------------
>>> Reporter: gjm | Owner: nobody
>>> Type: task | Status: new
>>> Priority: critical | Milestone: Release 2
>>> Component: siteadmin | Version:
>>> Resolution: | Keywords:
>>> ------------------------+-----------------------
>>>
>>> Comment (by gjm):
>>>
>>> As part of my investigation around creating a mirror of the bloodhound
>>> portion, I have written a very simple script (only complicated when I
>>> decided to have a quick look at the argparse module and python2/3
>>> issues),
>>> inspired by a suggestion from Philip Martin.
>>>
>>> I have attached that script [attachment:emptyrevs.py here]. It may be
>>> worth putting in the bloodhound repository in case we need it again of
>>> course.
>>>
>>
>
--
Dave Brondsema : dave@brondsema.net
http://www.brondsema.net : personal
http://www.splike.com : programming
<><
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache repo for browse functionality
Posted by Jim Jagielski <ji...@jaguNET.com>.
We can simply re-ask infra... :)
On Aug 9, 2012, at 5:13 AM, Greg Stein <gs...@gmail.com> wrote:
> Looks like the Bloodhound guys might be amenable to our use of their instance.
>
> That said: Infra has already constructed the BZ stuff that Jim asked for. :-/
>
> -g
>
> ---------- Forwarded message ----------
> From: Gary Martin <ga...@wandisco.com>
> Date: Thu, Aug 9, 2012 at 5:01 AM
> Subject: Re: [Apache Bloodhound] #156: Local copy of bloodhound part
> of Apache repo for browse functionality
> To: bloodhound-dev@incubator.apache.org
>
>
> On 08/09/2012 06:52 AM, Greg Stein wrote:
>>
>> On Thu, Aug 9, 2012 at 1:27 AM, Olemis Lang <ol...@gmail.com> wrote:
>>>
>>> ...
>>> Well . I don't know if I understood correctly , so it's not clear to
>>> me whether we will have git or svn repository after all this . In case
>>
>> We'll have a copy of the primary ASF repository (ie. .../repos/asf/...).
>>
>> My comments regarding Git repositories were for the Allura people. For
>> Bloodhound to be fully usable at the ASF, we'll want Git support at
>> some point, but that can certainly wait.
>>
>> (on that note, the new Apache Steve TLP (no web site yet; small group
>> to do STV voting tools) is pondering asking whether they could use BH,
>> and whether this community would support that)
>
>
> That would certainly be cool with me. Hopefully we will get
> https://issues.apache.org/bloodhound/ updated soon to demonstrate how
> we are doing. Perhaps that would make it easier for Apache Steve to
> decide. There are things that would be nice for us to improve on but,
> as far as I can tell, we have not stopped from being a usable system.
> Anyway, anything that drives additional feedback would be fantastic.
>
>
>>> we are talking of git , the only thing I wanted to mention is that
>>> there are some (performance) issues regarding git support in Trac .
>>
>> No worries. Something for another day.
>
>
> Yes, interesting that. I will look into it soon I hope. Thanks for the
> heads up, Olemis.
>
> Cheers,
> Gary
>
Fwd: [Apache Bloodhound] #156: Local copy of bloodhound part of
Apache repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
Looks like the Bloodhound guys might be amenable to our use of their instance.
That said: Infra has already constructed the BZ stuff that Jim asked for. :-/
-g
---------- Forwarded message ----------
From: Gary Martin <ga...@wandisco.com>
Date: Thu, Aug 9, 2012 at 5:01 AM
Subject: Re: [Apache Bloodhound] #156: Local copy of bloodhound part
of Apache repo for browse functionality
To: bloodhound-dev@incubator.apache.org
On 08/09/2012 06:52 AM, Greg Stein wrote:
>
> On Thu, Aug 9, 2012 at 1:27 AM, Olemis Lang <ol...@gmail.com> wrote:
>>
>> ...
>> Well . I don't know if I understood correctly , so it's not clear to
>> me whether we will have git or svn repository after all this . In case
>
> We'll have a copy of the primary ASF repository (ie. .../repos/asf/...).
>
> My comments regarding Git repositories were for the Allura people. For
> Bloodhound to be fully usable at the ASF, we'll want Git support at
> some point, but that can certainly wait.
>
> (on that note, the new Apache Steve TLP (no web site yet; small group
> to do STV voting tools) is pondering asking whether they could use BH,
> and whether this community would support that)
That would certainly be cool with me. Hopefully we will get
https://issues.apache.org/bloodhound/ updated soon to demonstrate how
we are doing. Perhaps that would make it easier for Apache Steve to
decide. There are things that would be nice for us to improve on but,
as far as I can tell, we have not stopped from being a usable system.
Anyway, anything that drives additional feedback would be fantastic.
>> we are talking of git , the only thing I wanted to mention is that
>> there are some (performance) issues regarding git support in Trac .
>
> No worries. Something for another day.
Yes, interesting that. I will look into it soon I hope. Thanks for the
heads up, Olemis.
Cheers,
Gary
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Olemis Lang <ol...@gmail.com>.
On 8/9/12, Gary Martin <ga...@wandisco.com> wrote:
> On 08/09/2012 06:52 AM, Greg Stein wrote:
>> On Thu, Aug 9, 2012 at 1:27 AM, Olemis Lang <ol...@gmail.com> wrote:
>>> ...
[...]
>>
>> (on that note, the new Apache Steve TLP (no web site yet; small group
>> to do STV voting tools) is pondering asking whether they could use BH,
>> and whether this community would support that)
>
> That would certainly be cool with me.
I second that ...
[...]
> There are things that would be nice for us to improve on
yep. The change will be more evident after Release 3 .
> but, as far as
> I can tell, we have not stopped from being a usable system. Anyway,
> anything that drives additional feedback would be fantastic.
>
definitely sure !
>>> we are talking of git , the only thing I wanted to mention is that
>>> there are some (performance) issues regarding git support in Trac .
>> No worries. Something for another day.
>
> Yes, interesting that. I will look into it soon I hope. Thanks for the
> heads up, Olemis.
>
https://groups.google.com/forum/?fromgroups#!topic/trac-dev/UcbwUjwCM7Q
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
On 08/09/2012 06:52 AM, Greg Stein wrote:
> On Thu, Aug 9, 2012 at 1:27 AM, Olemis Lang <ol...@gmail.com> wrote:
>> ...
>> Well . I don't know if I understood correctly , so it's not clear to
>> me whether we will have git or svn repository after all this . In case
> We'll have a copy of the primary ASF repository (ie. .../repos/asf/...).
>
> My comments regarding Git repositories were for the Allura people. For
> Bloodhound to be fully usable at the ASF, we'll want Git support at
> some point, but that can certainly wait.
>
> (on that note, the new Apache Steve TLP (no web site yet; small group
> to do STV voting tools) is pondering asking whether they could use BH,
> and whether this community would support that)
That would certainly be cool with me. Hopefully we will get
https://issues.apache.org/bloodhound/ updated soon to demonstrate how we
are doing. Perhaps that would make it easier for Apache Steve to decide.
There are things that would be nice for us to improve on but, as far as
I can tell, we have not stopped from being a usable system. Anyway,
anything that drives additional feedback would be fantastic.
>> we are talking of git , the only thing I wanted to mention is that
>> there are some (performance) issues regarding git support in Trac .
> No worries. Something for another day.
Yes, interesting that. I will look into it soon I hope. Thanks for the
heads up, Olemis.
Cheers,
Gary
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Aug 9, 2012 at 1:27 AM, Olemis Lang <ol...@gmail.com> wrote:
>...
> Well . I don't know if I understood correctly , so it's not clear to
> me whether we will have git or svn repository after all this . In case
We'll have a copy of the primary ASF repository (ie. .../repos/asf/...).
My comments regarding Git repositories were for the Allura people. For
Bloodhound to be fully usable at the ASF, we'll want Git support at
some point, but that can certainly wait.
(on that note, the new Apache Steve TLP (no web site yet; small group
to do STV voting tools) is pondering asking whether they could use BH,
and whether this community would support that)
> we are talking of git , the only thing I wanted to mention is that
> there are some (performance) issues regarding git support in Trac .
No worries. Something for another day.
>...
Cheers,
-g
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Olemis Lang <ol...@gmail.com>.
On 8/8/12, Greg Stein <gs...@gmail.com> wrote:
> Very cool!
>
> But with that said... I was on IRC, and the Infra guys might actually
> create a full repository mirror for us. The thing is that Apache
> Allura (also incubating) will want a copy of their project(s), too.
> Tho... I think they're going with Git, but the concept is the same.
> Bloodhound and Allura both need local read-access to repositories.
>
[...]
>
> I believe the Right Answer here is for both projects to confirm that
> this option is workable, and for us to ask Infra to start setting it
> up.
>
> Thoughts?
>
Well . I don't know if I understood correctly , so it's not clear to
me whether we will have git or svn repository after all this . In case
we are talking of git , the only thing I wanted to mention is that
there are some (performance) issues regarding git support in Trac .
There's a recent thread about that subject in trac-dev ML . If this is
something to consider then I can post a link to that thread for
further details .
:-/
If it's an svn repos in the end then ... nothing to say ...
:-$
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
I assumed that you would be able to confirm that!
Actually, this approach could be very helpful as I was worried about
having to sort out the python bindings for subversion 1.7 for the
version of python that is used on the VM. If I don't need svnrdump, I
should be able to use the standard distro packages.
I assume that this is of less interest for the allura-dev list so sorry
if this is a bit off topic for you.
Cheers,
Gary
On 08/08/12 23:40, Greg Stein wrote:
>
> Subversion has no problems at all with NFS.
>
> On Aug 8, 2012 6:36 PM, "Gary Martin" <gary.martin@wandisco.com
> <ma...@wandisco.com>> wrote:
>
> Hi everyone,
>
> I should work out how to test this suggestion but I would expect
> it to work. I got Bloodhound set up with the postgresql backend so
> that aspect should be fine (I seem to remember reading about
> problems when using sqlite). After that, I assume that we can work
> with everything as read-only from bloodhound's point of view.
>
> So, as long as subversion can also play nicely with NFS then I
> have no current reason to suspect problems with this approach at
> this point.
>
> Cheers,
> Gary
>
>
> On 08/08/12 22:26, Greg Stein wrote:
>
> Very cool!
>
> But with that said... I was on IRC, and the Infra guys might
> actually
> create a full repository mirror for us. The thing is that Apache
> Allura (also incubating) will want a copy of their project(s),
> too.
> Tho... I think they're going with Git, but the concept is the
> same.
> Bloodhound and Allura both need local read-access to repositories.
>
> Infra was thinking about sync'ing repository copies over to a
> box (I
> forget the name). The BH and Allura VMs would be migrated over
> to that
> same box. The repositories would then be exposed via NFS, and
> the two
> VMs would (locally) mount that NFS share.
>
> I believe the Right Answer here is for both projects to
> confirm that
> this option is workable, and for us to ask Infra to start
> setting it
> up.
>
> Thoughts?
>
> Cheers,
> -g
>
> On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin
> <gary.martin@wandisco.com <ma...@wandisco.com>>
> wrote:
>
> Not sure if this could be of interest beyond Bloodhound.
> It is a very simple
> script (and quite possibly over-complicated in reality).
>
> The attachment referred to is here:
> https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
>
> Just tested it on 1229640 empty commits:
>
> python3 emptyrevs.py 1229640 | eatmydata svnadmin
> --bypass-prop-validation load repos/
>
> which took about 2 hours - I suspect that the svnadmin
> load was the real
> bottleneck.. it is just a few seconds to create about 92M
> of data if you
> direct to a file instead.
>
> Cheers,
> Gary
>
>
>
> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>
> #156: Local copy of bloodhound part of Apache repo for
> browse
> functionality
> ------------------------+-----------------------
> Reporter: gjm | Owner: nobody
> Type: task | Status: new
> Priority: critical | Milestone: Release 2
> Component: siteadmin | Version:
> Resolution: | Keywords:
> ------------------------+-----------------------
>
> Comment (by gjm):
>
> As part of my investigation around creating a
> mirror of the bloodhound
> portion, I have written a very simple script (only
> complicated when I
> decided to have a quick look at the argparse module
> and python2/3
> issues),
> inspired by a suggestion from Philip Martin.
>
> I have attached that script
> [attachment:emptyrevs.py here]. It may be
> worth putting in the bloodhound repository in case
> we need it again of
> course.
>
>
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
I assumed that you would be able to confirm that!
Actually, this approach could be very helpful as I was worried about
having to sort out the python bindings for subversion 1.7 for the
version of python that is used on the VM. If I don't need svnrdump, I
should be able to use the standard distro packages.
I assume that this is of less interest for the allura-dev list so sorry
if this is a bit off topic for you.
Cheers,
Gary
On 08/08/12 23:40, Greg Stein wrote:
>
> Subversion has no problems at all with NFS.
>
> On Aug 8, 2012 6:36 PM, "Gary Martin" <gary.martin@wandisco.com
> <ma...@wandisco.com>> wrote:
>
> Hi everyone,
>
> I should work out how to test this suggestion but I would expect
> it to work. I got Bloodhound set up with the postgresql backend so
> that aspect should be fine (I seem to remember reading about
> problems when using sqlite). After that, I assume that we can work
> with everything as read-only from bloodhound's point of view.
>
> So, as long as subversion can also play nicely with NFS then I
> have no current reason to suspect problems with this approach at
> this point.
>
> Cheers,
> Gary
>
>
> On 08/08/12 22:26, Greg Stein wrote:
>
> Very cool!
>
> But with that said... I was on IRC, and the Infra guys might
> actually
> create a full repository mirror for us. The thing is that Apache
> Allura (also incubating) will want a copy of their project(s),
> too.
> Tho... I think they're going with Git, but the concept is the
> same.
> Bloodhound and Allura both need local read-access to repositories.
>
> Infra was thinking about sync'ing repository copies over to a
> box (I
> forget the name). The BH and Allura VMs would be migrated over
> to that
> same box. The repositories would then be exposed via NFS, and
> the two
> VMs would (locally) mount that NFS share.
>
> I believe the Right Answer here is for both projects to
> confirm that
> this option is workable, and for us to ask Infra to start
> setting it
> up.
>
> Thoughts?
>
> Cheers,
> -g
>
> On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin
> <gary.martin@wandisco.com <ma...@wandisco.com>>
> wrote:
>
> Not sure if this could be of interest beyond Bloodhound.
> It is a very simple
> script (and quite possibly over-complicated in reality).
>
> The attachment referred to is here:
> https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
>
> Just tested it on 1229640 empty commits:
>
> python3 emptyrevs.py 1229640 | eatmydata svnadmin
> --bypass-prop-validation load repos/
>
> which took about 2 hours - I suspect that the svnadmin
> load was the real
> bottleneck.. it is just a few seconds to create about 92M
> of data if you
> direct to a file instead.
>
> Cheers,
> Gary
>
>
>
> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>
> #156: Local copy of bloodhound part of Apache repo for
> browse
> functionality
> ------------------------+-----------------------
> Reporter: gjm | Owner: nobody
> Type: task | Status: new
> Priority: critical | Milestone: Release 2
> Component: siteadmin | Version:
> Resolution: | Keywords:
> ------------------------+-----------------------
>
> Comment (by gjm):
>
> As part of my investigation around creating a
> mirror of the bloodhound
> portion, I have written a very simple script (only
> complicated when I
> decided to have a quick look at the argparse module
> and python2/3
> issues),
> inspired by a suggestion from Philip Martin.
>
> I have attached that script
> [attachment:emptyrevs.py here]. It may be
> worth putting in the bloodhound repository in case
> we need it again of
> course.
>
>
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
Subversion has no problems at all with NFS.
On Aug 8, 2012 6:36 PM, "Gary Martin" <ga...@wandisco.com> wrote:
> Hi everyone,
>
> I should work out how to test this suggestion but I would expect it to
> work. I got Bloodhound set up with the postgresql backend so that aspect
> should be fine (I seem to remember reading about problems when using
> sqlite). After that, I assume that we can work with everything as read-only
> from bloodhound's point of view.
>
> So, as long as subversion can also play nicely with NFS then I have no
> current reason to suspect problems with this approach at this point.
>
> Cheers,
> Gary
>
>
> On 08/08/12 22:26, Greg Stein wrote:
>
>> Very cool!
>>
>> But with that said... I was on IRC, and the Infra guys might actually
>> create a full repository mirror for us. The thing is that Apache
>> Allura (also incubating) will want a copy of their project(s), too.
>> Tho... I think they're going with Git, but the concept is the same.
>> Bloodhound and Allura both need local read-access to repositories.
>>
>> Infra was thinking about sync'ing repository copies over to a box (I
>> forget the name). The BH and Allura VMs would be migrated over to that
>> same box. The repositories would then be exposed via NFS, and the two
>> VMs would (locally) mount that NFS share.
>>
>> I believe the Right Answer here is for both projects to confirm that
>> this option is workable, and for us to ask Infra to start setting it
>> up.
>>
>> Thoughts?
>>
>> Cheers,
>> -g
>>
>> On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin <ga...@wandisco.com>
>> wrote:
>>
>>> Not sure if this could be of interest beyond Bloodhound. It is a very
>>> simple
>>> script (and quite possibly over-complicated in reality).
>>>
>>> The attachment referred to is here:
>>> https://issues.apache.org/**bloodhound/attachment/ticket/**
>>> 156/emptyrevs.py<https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py>
>>>
>>> Just tested it on 1229640 empty commits:
>>>
>>> python3 emptyrevs.py 1229640 | eatmydata svnadmin
>>> --bypass-prop-validation load repos/
>>>
>>> which took about 2 hours - I suspect that the svnadmin load was the real
>>> bottleneck.. it is just a few seconds to create about 92M of data if you
>>> direct to a file instead.
>>>
>>> Cheers,
>>> Gary
>>>
>>>
>>>
>>> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>>>
>>>> #156: Local copy of bloodhound part of Apache repo for browse
>>>> functionality
>>>> ------------------------+-----**------------------
>>>> Reporter: gjm | Owner: nobody
>>>> Type: task | Status: new
>>>> Priority: critical | Milestone: Release 2
>>>> Component: siteadmin | Version:
>>>> Resolution: | Keywords:
>>>> ------------------------+-----**------------------
>>>>
>>>> Comment (by gjm):
>>>>
>>>> As part of my investigation around creating a mirror of the
>>>> bloodhound
>>>> portion, I have written a very simple script (only complicated when I
>>>> decided to have a quick look at the argparse module and python2/3
>>>> issues),
>>>> inspired by a suggestion from Philip Martin.
>>>>
>>>> I have attached that script [attachment:emptyrevs.py here]. It may be
>>>> worth putting in the bloodhound repository in case we need it again
>>>> of
>>>> course.
>>>>
>>>>
>
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
Subversion has no problems at all with NFS.
On Aug 8, 2012 6:36 PM, "Gary Martin" <ga...@wandisco.com> wrote:
> Hi everyone,
>
> I should work out how to test this suggestion but I would expect it to
> work. I got Bloodhound set up with the postgresql backend so that aspect
> should be fine (I seem to remember reading about problems when using
> sqlite). After that, I assume that we can work with everything as read-only
> from bloodhound's point of view.
>
> So, as long as subversion can also play nicely with NFS then I have no
> current reason to suspect problems with this approach at this point.
>
> Cheers,
> Gary
>
>
> On 08/08/12 22:26, Greg Stein wrote:
>
>> Very cool!
>>
>> But with that said... I was on IRC, and the Infra guys might actually
>> create a full repository mirror for us. The thing is that Apache
>> Allura (also incubating) will want a copy of their project(s), too.
>> Tho... I think they're going with Git, but the concept is the same.
>> Bloodhound and Allura both need local read-access to repositories.
>>
>> Infra was thinking about sync'ing repository copies over to a box (I
>> forget the name). The BH and Allura VMs would be migrated over to that
>> same box. The repositories would then be exposed via NFS, and the two
>> VMs would (locally) mount that NFS share.
>>
>> I believe the Right Answer here is for both projects to confirm that
>> this option is workable, and for us to ask Infra to start setting it
>> up.
>>
>> Thoughts?
>>
>> Cheers,
>> -g
>>
>> On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin <ga...@wandisco.com>
>> wrote:
>>
>>> Not sure if this could be of interest beyond Bloodhound. It is a very
>>> simple
>>> script (and quite possibly over-complicated in reality).
>>>
>>> The attachment referred to is here:
>>> https://issues.apache.org/**bloodhound/attachment/ticket/**
>>> 156/emptyrevs.py<https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py>
>>>
>>> Just tested it on 1229640 empty commits:
>>>
>>> python3 emptyrevs.py 1229640 | eatmydata svnadmin
>>> --bypass-prop-validation load repos/
>>>
>>> which took about 2 hours - I suspect that the svnadmin load was the real
>>> bottleneck.. it is just a few seconds to create about 92M of data if you
>>> direct to a file instead.
>>>
>>> Cheers,
>>> Gary
>>>
>>>
>>>
>>> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>>>
>>>> #156: Local copy of bloodhound part of Apache repo for browse
>>>> functionality
>>>> ------------------------+-----**------------------
>>>> Reporter: gjm | Owner: nobody
>>>> Type: task | Status: new
>>>> Priority: critical | Milestone: Release 2
>>>> Component: siteadmin | Version:
>>>> Resolution: | Keywords:
>>>> ------------------------+-----**------------------
>>>>
>>>> Comment (by gjm):
>>>>
>>>> As part of my investigation around creating a mirror of the
>>>> bloodhound
>>>> portion, I have written a very simple script (only complicated when I
>>>> decided to have a quick look at the argparse module and python2/3
>>>> issues),
>>>> inspired by a suggestion from Philip Martin.
>>>>
>>>> I have attached that script [attachment:emptyrevs.py here]. It may be
>>>> worth putting in the bloodhound repository in case we need it again
>>>> of
>>>> course.
>>>>
>>>>
>
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
Hi everyone,
I should work out how to test this suggestion but I would expect it to
work. I got Bloodhound set up with the postgresql backend so that aspect
should be fine (I seem to remember reading about problems when using
sqlite). After that, I assume that we can work with everything as
read-only from bloodhound's point of view.
So, as long as subversion can also play nicely with NFS then I have no
current reason to suspect problems with this approach at this point.
Cheers,
Gary
On 08/08/12 22:26, Greg Stein wrote:
> Very cool!
>
> But with that said... I was on IRC, and the Infra guys might actually
> create a full repository mirror for us. The thing is that Apache
> Allura (also incubating) will want a copy of their project(s), too.
> Tho... I think they're going with Git, but the concept is the same.
> Bloodhound and Allura both need local read-access to repositories.
>
> Infra was thinking about sync'ing repository copies over to a box (I
> forget the name). The BH and Allura VMs would be migrated over to that
> same box. The repositories would then be exposed via NFS, and the two
> VMs would (locally) mount that NFS share.
>
> I believe the Right Answer here is for both projects to confirm that
> this option is workable, and for us to ask Infra to start setting it
> up.
>
> Thoughts?
>
> Cheers,
> -g
>
> On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin <ga...@wandisco.com> wrote:
>> Not sure if this could be of interest beyond Bloodhound. It is a very simple
>> script (and quite possibly over-complicated in reality).
>>
>> The attachment referred to is here:
>> https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
>>
>> Just tested it on 1229640 empty commits:
>>
>> python3 emptyrevs.py 1229640 | eatmydata svnadmin
>> --bypass-prop-validation load repos/
>>
>> which took about 2 hours - I suspect that the svnadmin load was the real
>> bottleneck.. it is just a few seconds to create about 92M of data if you
>> direct to a file instead.
>>
>> Cheers,
>> Gary
>>
>>
>>
>> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>>> #156: Local copy of bloodhound part of Apache repo for browse
>>> functionality
>>> ------------------------+-----------------------
>>> Reporter: gjm | Owner: nobody
>>> Type: task | Status: new
>>> Priority: critical | Milestone: Release 2
>>> Component: siteadmin | Version:
>>> Resolution: | Keywords:
>>> ------------------------+-----------------------
>>>
>>> Comment (by gjm):
>>>
>>> As part of my investigation around creating a mirror of the bloodhound
>>> portion, I have written a very simple script (only complicated when I
>>> decided to have a quick look at the argparse module and python2/3
>>> issues),
>>> inspired by a suggestion from Philip Martin.
>>>
>>> I have attached that script [attachment:emptyrevs.py here]. It may be
>>> worth putting in the bloodhound repository in case we need it again of
>>> course.
>>>
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Dave Brondsema <da...@brondsema.net>.
That sounds great. AFAIK an NFS mount will be fine for Allura to read from.
By the way, I've subscribed to the bloodhound-dev list also. It seems
that allura and bloodhound have a lot in common: python code/project
management tools both going through incubation :)
-Dave
On 8/8/12 5:26 PM, Greg Stein wrote:
> Very cool!
>
> But with that said... I was on IRC, and the Infra guys might actually
> create a full repository mirror for us. The thing is that Apache
> Allura (also incubating) will want a copy of their project(s), too.
> Tho... I think they're going with Git, but the concept is the same.
> Bloodhound and Allura both need local read-access to repositories.
>
> Infra was thinking about sync'ing repository copies over to a box (I
> forget the name). The BH and Allura VMs would be migrated over to that
> same box. The repositories would then be exposed via NFS, and the two
> VMs would (locally) mount that NFS share.
>
> I believe the Right Answer here is for both projects to confirm that
> this option is workable, and for us to ask Infra to start setting it
> up.
>
> Thoughts?
>
> Cheers,
> -g
>
> On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin <ga...@wandisco.com> wrote:
>> Not sure if this could be of interest beyond Bloodhound. It is a very simple
>> script (and quite possibly over-complicated in reality).
>>
>> The attachment referred to is here:
>> https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
>>
>> Just tested it on 1229640 empty commits:
>>
>> python3 emptyrevs.py 1229640 | eatmydata svnadmin
>> --bypass-prop-validation load repos/
>>
>> which took about 2 hours - I suspect that the svnadmin load was the real
>> bottleneck.. it is just a few seconds to create about 92M of data if you
>> direct to a file instead.
>>
>> Cheers,
>> Gary
>>
>>
>>
>> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>>>
>>> #156: Local copy of bloodhound part of Apache repo for browse
>>> functionality
>>> ------------------------+-----------------------
>>> Reporter: gjm | Owner: nobody
>>> Type: task | Status: new
>>> Priority: critical | Milestone: Release 2
>>> Component: siteadmin | Version:
>>> Resolution: | Keywords:
>>> ------------------------+-----------------------
>>>
>>> Comment (by gjm):
>>>
>>> As part of my investigation around creating a mirror of the bloodhound
>>> portion, I have written a very simple script (only complicated when I
>>> decided to have a quick look at the argparse module and python2/3
>>> issues),
>>> inspired by a suggestion from Philip Martin.
>>>
>>> I have attached that script [attachment:emptyrevs.py here]. It may be
>>> worth putting in the bloodhound repository in case we need it again of
>>> course.
>>>
>>
>
--
Dave Brondsema : dave@brondsema.net
http://www.brondsema.net : personal
http://www.splike.com : programming
<><
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Gary Martin <ga...@wandisco.com>.
Hi everyone,
I should work out how to test this suggestion but I would expect it to
work. I got Bloodhound set up with the postgresql backend so that aspect
should be fine (I seem to remember reading about problems when using
sqlite). After that, I assume that we can work with everything as
read-only from bloodhound's point of view.
So, as long as subversion can also play nicely with NFS then I have no
current reason to suspect problems with this approach at this point.
Cheers,
Gary
On 08/08/12 22:26, Greg Stein wrote:
> Very cool!
>
> But with that said... I was on IRC, and the Infra guys might actually
> create a full repository mirror for us. The thing is that Apache
> Allura (also incubating) will want a copy of their project(s), too.
> Tho... I think they're going with Git, but the concept is the same.
> Bloodhound and Allura both need local read-access to repositories.
>
> Infra was thinking about sync'ing repository copies over to a box (I
> forget the name). The BH and Allura VMs would be migrated over to that
> same box. The repositories would then be exposed via NFS, and the two
> VMs would (locally) mount that NFS share.
>
> I believe the Right Answer here is for both projects to confirm that
> this option is workable, and for us to ask Infra to start setting it
> up.
>
> Thoughts?
>
> Cheers,
> -g
>
> On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin <ga...@wandisco.com> wrote:
>> Not sure if this could be of interest beyond Bloodhound. It is a very simple
>> script (and quite possibly over-complicated in reality).
>>
>> The attachment referred to is here:
>> https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
>>
>> Just tested it on 1229640 empty commits:
>>
>> python3 emptyrevs.py 1229640 | eatmydata svnadmin
>> --bypass-prop-validation load repos/
>>
>> which took about 2 hours - I suspect that the svnadmin load was the real
>> bottleneck.. it is just a few seconds to create about 92M of data if you
>> direct to a file instead.
>>
>> Cheers,
>> Gary
>>
>>
>>
>> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>>> #156: Local copy of bloodhound part of Apache repo for browse
>>> functionality
>>> ------------------------+-----------------------
>>> Reporter: gjm | Owner: nobody
>>> Type: task | Status: new
>>> Priority: critical | Milestone: Release 2
>>> Component: siteadmin | Version:
>>> Resolution: | Keywords:
>>> ------------------------+-----------------------
>>>
>>> Comment (by gjm):
>>>
>>> As part of my investigation around creating a mirror of the bloodhound
>>> portion, I have written a very simple script (only complicated when I
>>> decided to have a quick look at the argparse module and python2/3
>>> issues),
>>> inspired by a suggestion from Philip Martin.
>>>
>>> I have attached that script [attachment:emptyrevs.py here]. It may be
>>> worth putting in the bloodhound repository in case we need it again of
>>> course.
>>>
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
Very cool!
But with that said... I was on IRC, and the Infra guys might actually
create a full repository mirror for us. The thing is that Apache
Allura (also incubating) will want a copy of their project(s), too.
Tho... I think they're going with Git, but the concept is the same.
Bloodhound and Allura both need local read-access to repositories.
Infra was thinking about sync'ing repository copies over to a box (I
forget the name). The BH and Allura VMs would be migrated over to that
same box. The repositories would then be exposed via NFS, and the two
VMs would (locally) mount that NFS share.
I believe the Right Answer here is for both projects to confirm that
this option is workable, and for us to ask Infra to start setting it
up.
Thoughts?
Cheers,
-g
On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin <ga...@wandisco.com> wrote:
> Not sure if this could be of interest beyond Bloodhound. It is a very simple
> script (and quite possibly over-complicated in reality).
>
> The attachment referred to is here:
> https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
>
> Just tested it on 1229640 empty commits:
>
> python3 emptyrevs.py 1229640 | eatmydata svnadmin
> --bypass-prop-validation load repos/
>
> which took about 2 hours - I suspect that the svnadmin load was the real
> bottleneck.. it is just a few seconds to create about 92M of data if you
> direct to a file instead.
>
> Cheers,
> Gary
>
>
>
> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>>
>> #156: Local copy of bloodhound part of Apache repo for browse
>> functionality
>> ------------------------+-----------------------
>> Reporter: gjm | Owner: nobody
>> Type: task | Status: new
>> Priority: critical | Milestone: Release 2
>> Component: siteadmin | Version:
>> Resolution: | Keywords:
>> ------------------------+-----------------------
>>
>> Comment (by gjm):
>>
>> As part of my investigation around creating a mirror of the bloodhound
>> portion, I have written a very simple script (only complicated when I
>> decided to have a quick look at the argparse module and python2/3
>> issues),
>> inspired by a suggestion from Philip Martin.
>>
>> I have attached that script [attachment:emptyrevs.py here]. It may be
>> worth putting in the bloodhound repository in case we need it again of
>> course.
>>
>
Re: [Apache Bloodhound] #156: Local copy of bloodhound part of Apache
repo for browse functionality
Posted by Greg Stein <gs...@gmail.com>.
Very cool!
But with that said... I was on IRC, and the Infra guys might actually
create a full repository mirror for us. The thing is that Apache
Allura (also incubating) will want a copy of their project(s), too.
Tho... I think they're going with Git, but the concept is the same.
Bloodhound and Allura both need local read-access to repositories.
Infra was thinking about sync'ing repository copies over to a box (I
forget the name). The BH and Allura VMs would be migrated over to that
same box. The repositories would then be exposed via NFS, and the two
VMs would (locally) mount that NFS share.
I believe the Right Answer here is for both projects to confirm that
this option is workable, and for us to ask Infra to start setting it
up.
Thoughts?
Cheers,
-g
On Tue, Aug 7, 2012 at 2:29 PM, Gary Martin <ga...@wandisco.com> wrote:
> Not sure if this could be of interest beyond Bloodhound. It is a very simple
> script (and quite possibly over-complicated in reality).
>
> The attachment referred to is here:
> https://issues.apache.org/bloodhound/attachment/ticket/156/emptyrevs.py
>
> Just tested it on 1229640 empty commits:
>
> python3 emptyrevs.py 1229640 | eatmydata svnadmin
> --bypass-prop-validation load repos/
>
> which took about 2 hours - I suspect that the svnadmin load was the real
> bottleneck.. it is just a few seconds to create about 92M of data if you
> direct to a file instead.
>
> Cheers,
> Gary
>
>
>
> On 08/07/2012 06:47 PM, Apache Bloodhound wrote:
>>
>> #156: Local copy of bloodhound part of Apache repo for browse
>> functionality
>> ------------------------+-----------------------
>> Reporter: gjm | Owner: nobody
>> Type: task | Status: new
>> Priority: critical | Milestone: Release 2
>> Component: siteadmin | Version:
>> Resolution: | Keywords:
>> ------------------------+-----------------------
>>
>> Comment (by gjm):
>>
>> As part of my investigation around creating a mirror of the bloodhound
>> portion, I have written a very simple script (only complicated when I
>> decided to have a quick look at the argparse module and python2/3
>> issues),
>> inspired by a suggestion from Philip Martin.
>>
>> I have attached that script [attachment:emptyrevs.py here]. It may be
>> worth putting in the bloodhound repository in case we need it again of
>> course.
>>
>