You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whimsical.apache.org by sebb <se...@gmail.com> on 2019/11/23 16:18:01 UTC
Does Whimsy need to have a copy of Bills?
AFAICT Whimsy does not use its local checkout of Bills, so the directory
could be dropped.
Re: Does Whimsy need to have a copy of Bills?
Posted by Dave Fisher <wa...@comcast.net>.
Rough classes would be:
Secretary
Treasurer
Board
Foundation
The groupings look pretty obvious.
Regards,
Dave
Sent from my iPhone
> On Nov 23, 2019, at 5:53 PM, Sam Ruby <ru...@intertwingly.net> wrote:
>
> On Sat, Nov 23, 2019 at 8:30 PM Dave Fisher <wa...@comcast.net> wrote:
>>
>> It would be great to have a mapping between whimsy sections and svn paths, and allow whimsy developers to choose which svn paths are used for their current efforts.
>
> https://github.com/apache/whimsy/blob/master/repository.yml
> https://whimsy.apache.org/test/dataflow
>
> The latter is a work in progress. Any efforts to fill in the blanks
> would be appreciated
>
>> I’ve felt uncomfortable with all the information I have to download to work on whimsy.
>
> I'm envisioning adding a configuration option where one can list which
> of these paths to either include or exclude (at your choice) from
> updates.
>
> My thoughts are that things that are small don't cause much concern.
> Here is the current sizes of the svn checkouts, in megabytes:
>
> 19331 iclas
> 1313 cclas
> 823 grants
> 699 Bills
> 673 member_apps
> 59 foundation_board
> 53 Meetings
> 51 minutes
> 10 site-img
> 6 steve
> 6 comdev-foundation
> 5 officers
> 4 foundation
> 3 board
> 3 apmail_bin
> 3 apachecon
> 2 officers_historic
> 2 incubator-content
> 2 brandlist
> 1 templates
> 1 site-root
> 1 site-board
> 1 qmail_control
> 1 personnel-duties
> 1 officers_surveys
> 1 ldap-map
> 1 incubator-podlings
> 1 foundation_mentors
> 1 comdevtalks
> 1 attic-xdocs
>
>> Sent from my iPhone
>
> - Sam Ruby
>
>>>> On Nov 23, 2019, at 5:07 PM, Sam Ruby <ru...@intertwingly.net> wrote:
>>>
>>> On Sat, Nov 23, 2019 at 11:18 AM sebb <se...@gmail.com> wrote:
>>>>
>>>> AFAICT Whimsy does not use its local checkout of Bills, so the directory
>>>> could be dropped.
>>>
>>> Agreed.
>>>
>>> - Sam Ruby
>>
Re: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Sun, Nov 24, 2019 at 7:22 AM sebb <se...@gmail.com> wrote:
>
> On Sun, 24 Nov 2019 at 01:53, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > Here is the current sizes of the svn checkouts, in megabytes:
> >
> > 19331 iclas
> > 1313 cclas
> > 823 grants
> > 699 Bills
> > 673 member_apps
>
> I don't think any of the above are actually needed, except to check for
> existence.
Perhaps we could have a rake task do a svn checkout --depth empty on
these directories. Anybody who changes their mind later wants them
could do a rm -rf on the directory followed by a 'rake update'
- Sam Ruby
Re: Does Whimsy need to have a copy of Bills?
Posted by sebb <se...@gmail.com>.
Top posting:
FTR, the following are no longer needed as checkouts:
Bills
cclas
iclas
member_apps
These were the largest checkouts; the total is now 209M according to du -h.
I don't know if it's possible (necessary?) to prune any further.
It might be possible to split the remaining checkouts into essential and
optional groups.
On Tue, 26 Nov 2019 at 17:18, Sam Ruby <ru...@intertwingly.net> wrote:
> On Tue, Nov 26, 2019 at 12:06 PM sebb <se...@gmail.com> wrote:
> >
> > My takeaway is that using 'svn ls' without recursion is OK.
>
> +1
>
> > I've just done a check on member_apps and I get:
> >
> > $ time svn up member_apps/
> > Updating 'member_apps':
> > At revision 93890.
> >
> > real 0m1.573s
> > user 0m0.055s
> > sys 0m0.027s
> >
> > $ time svn ls https://svn.apache.org/repos/private/documents/member_apps
> |
> > wc -l
> > 882
> >
> > real 0m0.943s
> > user 0m0.053s
> > sys 0m0.026s
> >
> > i.e. svn update is ~ 50% slower than svn ls
>
> Something non-linear happens somewhere. svn update is actually
> (marginally) faster than svn ls on iclas:
>
> % time svn up iclas/
> Updating 'iclas':
> At revision 93890.
> svn up iclas/ 0.21s user 0.15s system 20% cpu 1.742 total
>
> % time svn ls https://svn.apache.org/repos/private/documents/iclas | wc -l
> 11038
> svn ls https://svn.apache.org/repos/private/documents/iclas 0.08s
> user 0.02s system 4% cpu 2.088 total
> wc -l 0.00s user 0.01s system 0% cpu 2.088 total
>
> > However, when using recursion, svn ls is quite a bit slower:
> >
> > $ time svn ls https://svn.apache.org/repos/private/documents/member_apps
> -R
> > | wc -l
> > 931
> >
> > real 0m3.161s
> > user 0m0.062s
> > sys 0m0.031s
>
> Again, the size of the directory makes a difference. Here's what I
> see for iclas:
>
> % time svn ls https://svn.apache.org/repos/private/documents/iclas -R |
> wc -l
> 12816
> svn ls https://svn.apache.org/repos/private/documents/iclas -R 0.22s
> user 0.07s system 0% cpu 1:04.50 total
> wc -l 0.00s user 0.01s system 0% cpu 1:04.50 total
>
> - Sam Ruby
>
> P.S. Just now discovering that time is built into zsh. Cool.
>
> > On Tue, 26 Nov 2019 at 00:36, Greg Stein <gs...@gmail.com> wrote:
> >
> > > One quick answer from earlier in the thread:
> > >
> > > > > > > What's most concerning is not just the elapsed time, but that
> this
> > > > > > > likely means that the call is expensive on the server, which
> may
> > > > > > > impact other users.
> > > > > > >
> > > > > > >
> > > > > > I see this as premature optimisation; we don't know whether svn
> list
> > > is
> > > > > > more expensive than svn update overall.
> > > > > > There may be other reasons why list is slower. Nor do we know if
> the
> > > > > > request will impact other users.
> > >
> > > "svn list" must generate the listing of 12k+ files, recursively. That
> takes
> > > some time to process and deliver over the network. I believe it is
> likely a
> > > PROPFIND which introduces some overheads on both ends (XML
> construction on
> > > server, parsing on client; sheer network size, too).
> > >
> > > "svn up" generates a diff report. "get these 3 files", and that's
> easily
> > > extracted from the difference between revision-working-copy and
> > > revision-server (plus some other concerns).
> > >
> > > So yes: an update is *way* faster, all around.
> > >
> > >
> > > On Mon, Nov 25, 2019 at 5:15 PM Sam Ruby <ru...@intertwingly.net>
> wrote:
> > >
> > > > Actually copy Greg this time.
> > > >
> > > > ---------- Forwarded message ---------
> > > > From: Sam Ruby <ru...@intertwingly.net>
> > > > Date: Mon, Nov 25, 2019, 4:15 PM
> > > > Subject: Re: Does Whimsy need to have a copy of Bills?
> > > > To: Whimsy dev <de...@whimsical.apache.org>
> > > >
> > > >
> > > > adding Greg to email.
> > > >
> > > > Recap: a change is being proposed whereas whimsy will do the
> > > > equivalent of the following command after every icla is processed:
> > > >
> > > > svn ls https://svn.apache.org/repos/private/documents/iclas --depth
> > > > infinity
> > > >
> > >
> > > Do you really need to use depth=infinity? The directory name is likely
> > > sufficient information. ?
> > >
> > > depth=immediates (the default for svn ls) is going to be just a few
> > > seconds.
> > >
> > > Currently this appears to take around forty to sixty elapsed seconds to
> > > > process.
> > > >
> > > > Questions for Greg:
> > > > 1) Does this proposed workload present an unreasonable load on the
> svn
> > > > server?
> > > >
> > >
> > > This should be fine as long as you don't put a "-v" switch in there.
> > > That'll take 10-15 minutes as it reconstructs all the files on the
> server
> > > and measures their size.
> > >
> > > The listing is just a single thread on the server, a fetch of the
> directory
> > > names, and then assembly/delivery of that result. There really
> shouldn't be
> > > any contention with other users, or heavy use of the CPU.
> > >
> > >
> > > > 2) Are there any faster alternatives which get us a list of names
> but no
> > > > data
> > > >
> > >
> > > So I experimented with a hack. I did a "full checkout" of the iclas
> > > directory, but stopped it after a single file was checked out. This
> left a
> > > partial checkout. Subversion will tell the server "I have $these. what
> am I
> > > missing?" when you run "svn status -u". You'll get a listing of the 12k
> > > missing files. Takes about 7 seconds or so.
> > >
> > > Specifically:
> > > $ # kill after reading/printing one line (the first file checked out)
> > > $ svn co https://svn.apache.org/repos/private/documents/iclas |
> python -c
> > > 'import signal,os,sys ; print sys.stdin.readline() ;
> > > os.killpg(os.getpgrp(), signal.SIGHUP)'
> > > A iclas/jiwei-guo
> > >
> > > svn: E200015: Caught signal
> > > svn: E200042: Additional errors:
> > > svn: E200015: Caught signal
> > > Hangup
> > > $ # get a recursive listing via status
> > > $ time svn st -u iclas | wc -l
> > > 12818
> > >
> > > real 0m6.782s
> > > user 0m3.461s
> > > sys 0m2.358s
> > >
> > > The status output should be easy to parse (it is designed as a
> fixed-width
> > > set of codes, then filename).
> > >
> > > Even if you do a full/normal checkout, note that "svn status -u" may be
> > > useful. Depending on whether you need the content, or just the names,
> you
> > > may want to migrate to the status-based approach.
> > >
> > > Oh! Just realized a better way, to avoid the hack/partial checkout.
> Even
> > > better, just check out the "iclas" directory for the revision it was
> > > created. It is an empty directory in that revision (a sibling directory
> > > received a bunch of Member applications, but those won't be in this
> > > checkout).
> > >
> > > $ svn -r 9696 co https://svn.apache.org/repos/private/documents/iclas
> > > Checked out revision 9696.
> > >
> > > The "svn status" works the same against the above (empty) working copy.
> > > Also at about 8 seconds.
> > >
> > > So. In summary, use "svn status" against a HEAD checkout, or against
> r9696
> > > for those who don't want the gigabytes of ICLA forms.
> > >
> > > A similar technique can be used for any of the other Whimsy data
> > > directories, of course. To find when a particular directory was
> created:
> > >
> > > $ # run an "svn log" in reverse order, and limit/stop at the first log
> > > entry.
> > > $ svn log --stop-on-copy --limit 1 -r0:HEAD
> > > https://svn.apache.org/repos/private/documents/iclas
> > >
> ------------------------------------------------------------------------
> > > r9696 | jim | 2006-11-17 10:35:28 -0600 (Fri, 17 Nov 2006) | 3 lines
> > >
> > > Start loading of scanned docs. Start with creating
> > > the dirs and upload the member apps
> > >
> > >
> ------------------------------------------------------------------------
> > >
> > > Note that sometimes a directory is created with content, in that
> revision.
> > > The "iclas" directory just happened to be created empty. But I imagine
> most
> > > directories will be much smaller at their creation, than they are
> today.
> > > (iow, don't expect them to always be empty at creation)
> > >
> > > Hope that helps,
> > > -g
> > >
> > >
> > >
> > > > - Sam Ruby
> > > >
> > > > On Mon, Nov 25, 2019 at 3:44 PM sebb <se...@gmail.com> wrote:
> > > > >
> > > > > On Mon, 25 Nov 2019 at 19:48, Sam Ruby <ru...@intertwingly.net>
> wrote:
> > > > >
> > > > > > On Mon, Nov 25, 2019 at 2:29 PM sebb <se...@gmail.com> wrote:
> > > > > > >
> > > > > > > On Mon, 25 Nov 2019 at 17:59, Sam Ruby <rubys@intertwingly.net
> >
> > > > wrote:
> > > > > > >
> > > > > > > > On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com>
> wrote:
> > > > > > > > >
> > > > > > > > > I was thinking of using svn ls to create a listing file
> which
> > > > would
> > > > > > be
> > > > > > > > > cached locally.
> > > > > > > >
> > > > > > > > Unfortunately, some observations (numbers below are
> approximate):
> > > > > > > >
> > > > > > > > svn up on a populated iclas directory: one second
> > > > > > > >
> > > > > > > > svn ls on iclas: two seconds, but only returns depth one
> > > > > > > >
> > > > > > > > svn ls on iclas --depth infinity: one minute
> > > > > > > >
> > > > > > > > What's most concerning is not just the elapsed time, but that
> > > this
> > > > > > > > likely means that the call is expensive on the server, which
> may
> > > > > > > > impact other users.
> > > > > > > >
> > > > > > > >
> > > > > > > I see this as premature optimisation; we don't know whether svn
> > > list
> > > > is
> > > > > > > more expensive than svn update overall.
> > > > > > > There may be other reasons why list is slower. Nor do we know
> if
> > > the
> > > > > > > request will impact other users.
> > > > > > >
> > > > > > > Besides, if the code checks SVN info first, it will only need
> to
> > > > fetch
> > > > > > the
> > > > > > > updated listing when there has been a change.
> > > > > > > Those directories are not busy.
> > > > > > >
> > > > > > > Furthermore, every time a new test installation is set up,
> there is
> > > > > > > definitely a large load on the server and network.
> > > > > > > This network load in particular must be orders of magnitude
> greater
> > > > than
> > > > > > > for a listing.
> > > > > >
> > > > > > Sorry for not being clear. I would be very concerned if
> whimsy-vm4
> > > > > > were invoking svn ls --depth infinity every 10 minutes as the
> current
> > > > > > cron job does.
> > > > >
> > > > >
> > > > > That would not be the case.
> > > > >
> > > > > The job would use 'svn info' on the remote repo and only fetch the
> > > > listing
> > > > > if necessary.
> > > > >
> > > > > For the repos in question, changes are rare.
> > > > >
> > > > >
> > > > > > Before any such change is deployed, it would be wise
> > > > > > for us to check both with the infrastructure team and the
> subversion
> > > > > > team (Greg likely can help with both).
> > > > > >
> > > > > > I'm less concerned about the overhead on development machines,
> and
> > > > > > there I suspect that most users would be happy with a svn
> checkout
> > > > > > --depth empty.
> > > > > >
> > > > > >
> > > > > This would not allow testing of the functions that need to know the
> > > list
> > > > of
> > > > > file names.
> > > > >
> > > > >
> > > > > > - Sam Ruby
> > > > > >
> > > >
> > >
>
Re: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Tue, Nov 26, 2019 at 12:06 PM sebb <se...@gmail.com> wrote:
>
> My takeaway is that using 'svn ls' without recursion is OK.
+1
> I've just done a check on member_apps and I get:
>
> $ time svn up member_apps/
> Updating 'member_apps':
> At revision 93890.
>
> real 0m1.573s
> user 0m0.055s
> sys 0m0.027s
>
> $ time svn ls https://svn.apache.org/repos/private/documents/member_apps |
> wc -l
> 882
>
> real 0m0.943s
> user 0m0.053s
> sys 0m0.026s
>
> i.e. svn update is ~ 50% slower than svn ls
Something non-linear happens somewhere. svn update is actually
(marginally) faster than svn ls on iclas:
% time svn up iclas/
Updating 'iclas':
At revision 93890.
svn up iclas/ 0.21s user 0.15s system 20% cpu 1.742 total
% time svn ls https://svn.apache.org/repos/private/documents/iclas | wc -l
11038
svn ls https://svn.apache.org/repos/private/documents/iclas 0.08s
user 0.02s system 4% cpu 2.088 total
wc -l 0.00s user 0.01s system 0% cpu 2.088 total
> However, when using recursion, svn ls is quite a bit slower:
>
> $ time svn ls https://svn.apache.org/repos/private/documents/member_apps -R
> | wc -l
> 931
>
> real 0m3.161s
> user 0m0.062s
> sys 0m0.031s
Again, the size of the directory makes a difference. Here's what I
see for iclas:
% time svn ls https://svn.apache.org/repos/private/documents/iclas -R | wc -l
12816
svn ls https://svn.apache.org/repos/private/documents/iclas -R 0.22s
user 0.07s system 0% cpu 1:04.50 total
wc -l 0.00s user 0.01s system 0% cpu 1:04.50 total
- Sam Ruby
P.S. Just now discovering that time is built into zsh. Cool.
> On Tue, 26 Nov 2019 at 00:36, Greg Stein <gs...@gmail.com> wrote:
>
> > One quick answer from earlier in the thread:
> >
> > > > > > What's most concerning is not just the elapsed time, but that this
> > > > > > likely means that the call is expensive on the server, which may
> > > > > > impact other users.
> > > > > >
> > > > > >
> > > > > I see this as premature optimisation; we don't know whether svn list
> > is
> > > > > more expensive than svn update overall.
> > > > > There may be other reasons why list is slower. Nor do we know if the
> > > > > request will impact other users.
> >
> > "svn list" must generate the listing of 12k+ files, recursively. That takes
> > some time to process and deliver over the network. I believe it is likely a
> > PROPFIND which introduces some overheads on both ends (XML construction on
> > server, parsing on client; sheer network size, too).
> >
> > "svn up" generates a diff report. "get these 3 files", and that's easily
> > extracted from the difference between revision-working-copy and
> > revision-server (plus some other concerns).
> >
> > So yes: an update is *way* faster, all around.
> >
> >
> > On Mon, Nov 25, 2019 at 5:15 PM Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > > Actually copy Greg this time.
> > >
> > > ---------- Forwarded message ---------
> > > From: Sam Ruby <ru...@intertwingly.net>
> > > Date: Mon, Nov 25, 2019, 4:15 PM
> > > Subject: Re: Does Whimsy need to have a copy of Bills?
> > > To: Whimsy dev <de...@whimsical.apache.org>
> > >
> > >
> > > adding Greg to email.
> > >
> > > Recap: a change is being proposed whereas whimsy will do the
> > > equivalent of the following command after every icla is processed:
> > >
> > > svn ls https://svn.apache.org/repos/private/documents/iclas --depth
> > > infinity
> > >
> >
> > Do you really need to use depth=infinity? The directory name is likely
> > sufficient information. ?
> >
> > depth=immediates (the default for svn ls) is going to be just a few
> > seconds.
> >
> > Currently this appears to take around forty to sixty elapsed seconds to
> > > process.
> > >
> > > Questions for Greg:
> > > 1) Does this proposed workload present an unreasonable load on the svn
> > > server?
> > >
> >
> > This should be fine as long as you don't put a "-v" switch in there.
> > That'll take 10-15 minutes as it reconstructs all the files on the server
> > and measures their size.
> >
> > The listing is just a single thread on the server, a fetch of the directory
> > names, and then assembly/delivery of that result. There really shouldn't be
> > any contention with other users, or heavy use of the CPU.
> >
> >
> > > 2) Are there any faster alternatives which get us a list of names but no
> > > data
> > >
> >
> > So I experimented with a hack. I did a "full checkout" of the iclas
> > directory, but stopped it after a single file was checked out. This left a
> > partial checkout. Subversion will tell the server "I have $these. what am I
> > missing?" when you run "svn status -u". You'll get a listing of the 12k
> > missing files. Takes about 7 seconds or so.
> >
> > Specifically:
> > $ # kill after reading/printing one line (the first file checked out)
> > $ svn co https://svn.apache.org/repos/private/documents/iclas | python -c
> > 'import signal,os,sys ; print sys.stdin.readline() ;
> > os.killpg(os.getpgrp(), signal.SIGHUP)'
> > A iclas/jiwei-guo
> >
> > svn: E200015: Caught signal
> > svn: E200042: Additional errors:
> > svn: E200015: Caught signal
> > Hangup
> > $ # get a recursive listing via status
> > $ time svn st -u iclas | wc -l
> > 12818
> >
> > real 0m6.782s
> > user 0m3.461s
> > sys 0m2.358s
> >
> > The status output should be easy to parse (it is designed as a fixed-width
> > set of codes, then filename).
> >
> > Even if you do a full/normal checkout, note that "svn status -u" may be
> > useful. Depending on whether you need the content, or just the names, you
> > may want to migrate to the status-based approach.
> >
> > Oh! Just realized a better way, to avoid the hack/partial checkout. Even
> > better, just check out the "iclas" directory for the revision it was
> > created. It is an empty directory in that revision (a sibling directory
> > received a bunch of Member applications, but those won't be in this
> > checkout).
> >
> > $ svn -r 9696 co https://svn.apache.org/repos/private/documents/iclas
> > Checked out revision 9696.
> >
> > The "svn status" works the same against the above (empty) working copy.
> > Also at about 8 seconds.
> >
> > So. In summary, use "svn status" against a HEAD checkout, or against r9696
> > for those who don't want the gigabytes of ICLA forms.
> >
> > A similar technique can be used for any of the other Whimsy data
> > directories, of course. To find when a particular directory was created:
> >
> > $ # run an "svn log" in reverse order, and limit/stop at the first log
> > entry.
> > $ svn log --stop-on-copy --limit 1 -r0:HEAD
> > https://svn.apache.org/repos/private/documents/iclas
> > ------------------------------------------------------------------------
> > r9696 | jim | 2006-11-17 10:35:28 -0600 (Fri, 17 Nov 2006) | 3 lines
> >
> > Start loading of scanned docs. Start with creating
> > the dirs and upload the member apps
> >
> > ------------------------------------------------------------------------
> >
> > Note that sometimes a directory is created with content, in that revision.
> > The "iclas" directory just happened to be created empty. But I imagine most
> > directories will be much smaller at their creation, than they are today.
> > (iow, don't expect them to always be empty at creation)
> >
> > Hope that helps,
> > -g
> >
> >
> >
> > > - Sam Ruby
> > >
> > > On Mon, Nov 25, 2019 at 3:44 PM sebb <se...@gmail.com> wrote:
> > > >
> > > > On Mon, 25 Nov 2019 at 19:48, Sam Ruby <ru...@intertwingly.net> wrote:
> > > >
> > > > > On Mon, Nov 25, 2019 at 2:29 PM sebb <se...@gmail.com> wrote:
> > > > > >
> > > > > > On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net>
> > > wrote:
> > > > > >
> > > > > > > On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > I was thinking of using svn ls to create a listing file which
> > > would
> > > > > be
> > > > > > > > cached locally.
> > > > > > >
> > > > > > > Unfortunately, some observations (numbers below are approximate):
> > > > > > >
> > > > > > > svn up on a populated iclas directory: one second
> > > > > > >
> > > > > > > svn ls on iclas: two seconds, but only returns depth one
> > > > > > >
> > > > > > > svn ls on iclas --depth infinity: one minute
> > > > > > >
> > > > > > > What's most concerning is not just the elapsed time, but that
> > this
> > > > > > > likely means that the call is expensive on the server, which may
> > > > > > > impact other users.
> > > > > > >
> > > > > > >
> > > > > > I see this as premature optimisation; we don't know whether svn
> > list
> > > is
> > > > > > more expensive than svn update overall.
> > > > > > There may be other reasons why list is slower. Nor do we know if
> > the
> > > > > > request will impact other users.
> > > > > >
> > > > > > Besides, if the code checks SVN info first, it will only need to
> > > fetch
> > > > > the
> > > > > > updated listing when there has been a change.
> > > > > > Those directories are not busy.
> > > > > >
> > > > > > Furthermore, every time a new test installation is set up, there is
> > > > > > definitely a large load on the server and network.
> > > > > > This network load in particular must be orders of magnitude greater
> > > than
> > > > > > for a listing.
> > > > >
> > > > > Sorry for not being clear. I would be very concerned if whimsy-vm4
> > > > > were invoking svn ls --depth infinity every 10 minutes as the current
> > > > > cron job does.
> > > >
> > > >
> > > > That would not be the case.
> > > >
> > > > The job would use 'svn info' on the remote repo and only fetch the
> > > listing
> > > > if necessary.
> > > >
> > > > For the repos in question, changes are rare.
> > > >
> > > >
> > > > > Before any such change is deployed, it would be wise
> > > > > for us to check both with the infrastructure team and the subversion
> > > > > team (Greg likely can help with both).
> > > > >
> > > > > I'm less concerned about the overhead on development machines, and
> > > > > there I suspect that most users would be happy with a svn checkout
> > > > > --depth empty.
> > > > >
> > > > >
> > > > This would not allow testing of the functions that need to know the
> > list
> > > of
> > > > file names.
> > > >
> > > >
> > > > > - Sam Ruby
> > > > >
> > >
> >
Re: Does Whimsy need to have a copy of Bills?
Posted by sebb <se...@gmail.com>.
My takeaway is that using 'svn ls' without recursion is OK.
I've just done a check on member_apps and I get:
$ time svn up member_apps/
Updating 'member_apps':
At revision 93890.
real 0m1.573s
user 0m0.055s
sys 0m0.027s
$ time svn ls https://svn.apache.org/repos/private/documents/member_apps |
wc -l
882
real 0m0.943s
user 0m0.053s
sys 0m0.026s
i.e. svn update is ~ 50% slower than svn ls
However, when using recursion, svn ls is quite a bit slower:
$ time svn ls https://svn.apache.org/repos/private/documents/member_apps -R
| wc -l
931
real 0m3.161s
user 0m0.062s
sys 0m0.031s
On Tue, 26 Nov 2019 at 00:36, Greg Stein <gs...@gmail.com> wrote:
> One quick answer from earlier in the thread:
>
> > > > > What's most concerning is not just the elapsed time, but that this
> > > > > likely means that the call is expensive on the server, which may
> > > > > impact other users.
> > > > >
> > > > >
> > > > I see this as premature optimisation; we don't know whether svn list
> is
> > > > more expensive than svn update overall.
> > > > There may be other reasons why list is slower. Nor do we know if the
> > > > request will impact other users.
>
> "svn list" must generate the listing of 12k+ files, recursively. That takes
> some time to process and deliver over the network. I believe it is likely a
> PROPFIND which introduces some overheads on both ends (XML construction on
> server, parsing on client; sheer network size, too).
>
> "svn up" generates a diff report. "get these 3 files", and that's easily
> extracted from the difference between revision-working-copy and
> revision-server (plus some other concerns).
>
> So yes: an update is *way* faster, all around.
>
>
> On Mon, Nov 25, 2019 at 5:15 PM Sam Ruby <ru...@intertwingly.net> wrote:
>
> > Actually copy Greg this time.
> >
> > ---------- Forwarded message ---------
> > From: Sam Ruby <ru...@intertwingly.net>
> > Date: Mon, Nov 25, 2019, 4:15 PM
> > Subject: Re: Does Whimsy need to have a copy of Bills?
> > To: Whimsy dev <de...@whimsical.apache.org>
> >
> >
> > adding Greg to email.
> >
> > Recap: a change is being proposed whereas whimsy will do the
> > equivalent of the following command after every icla is processed:
> >
> > svn ls https://svn.apache.org/repos/private/documents/iclas --depth
> > infinity
> >
>
> Do you really need to use depth=infinity? The directory name is likely
> sufficient information. ?
>
> depth=immediates (the default for svn ls) is going to be just a few
> seconds.
>
> Currently this appears to take around forty to sixty elapsed seconds to
> > process.
> >
> > Questions for Greg:
> > 1) Does this proposed workload present an unreasonable load on the svn
> > server?
> >
>
> This should be fine as long as you don't put a "-v" switch in there.
> That'll take 10-15 minutes as it reconstructs all the files on the server
> and measures their size.
>
> The listing is just a single thread on the server, a fetch of the directory
> names, and then assembly/delivery of that result. There really shouldn't be
> any contention with other users, or heavy use of the CPU.
>
>
> > 2) Are there any faster alternatives which get us a list of names but no
> > data
> >
>
> So I experimented with a hack. I did a "full checkout" of the iclas
> directory, but stopped it after a single file was checked out. This left a
> partial checkout. Subversion will tell the server "I have $these. what am I
> missing?" when you run "svn status -u". You'll get a listing of the 12k
> missing files. Takes about 7 seconds or so.
>
> Specifically:
> $ # kill after reading/printing one line (the first file checked out)
> $ svn co https://svn.apache.org/repos/private/documents/iclas | python -c
> 'import signal,os,sys ; print sys.stdin.readline() ;
> os.killpg(os.getpgrp(), signal.SIGHUP)'
> A iclas/jiwei-guo
>
> svn: E200015: Caught signal
> svn: E200042: Additional errors:
> svn: E200015: Caught signal
> Hangup
> $ # get a recursive listing via status
> $ time svn st -u iclas | wc -l
> 12818
>
> real 0m6.782s
> user 0m3.461s
> sys 0m2.358s
>
> The status output should be easy to parse (it is designed as a fixed-width
> set of codes, then filename).
>
> Even if you do a full/normal checkout, note that "svn status -u" may be
> useful. Depending on whether you need the content, or just the names, you
> may want to migrate to the status-based approach.
>
> Oh! Just realized a better way, to avoid the hack/partial checkout. Even
> better, just check out the "iclas" directory for the revision it was
> created. It is an empty directory in that revision (a sibling directory
> received a bunch of Member applications, but those won't be in this
> checkout).
>
> $ svn -r 9696 co https://svn.apache.org/repos/private/documents/iclas
> Checked out revision 9696.
>
> The "svn status" works the same against the above (empty) working copy.
> Also at about 8 seconds.
>
> So. In summary, use "svn status" against a HEAD checkout, or against r9696
> for those who don't want the gigabytes of ICLA forms.
>
> A similar technique can be used for any of the other Whimsy data
> directories, of course. To find when a particular directory was created:
>
> $ # run an "svn log" in reverse order, and limit/stop at the first log
> entry.
> $ svn log --stop-on-copy --limit 1 -r0:HEAD
> https://svn.apache.org/repos/private/documents/iclas
> ------------------------------------------------------------------------
> r9696 | jim | 2006-11-17 10:35:28 -0600 (Fri, 17 Nov 2006) | 3 lines
>
> Start loading of scanned docs. Start with creating
> the dirs and upload the member apps
>
> ------------------------------------------------------------------------
>
> Note that sometimes a directory is created with content, in that revision.
> The "iclas" directory just happened to be created empty. But I imagine most
> directories will be much smaller at their creation, than they are today.
> (iow, don't expect them to always be empty at creation)
>
> Hope that helps,
> -g
>
>
>
> > - Sam Ruby
> >
> > On Mon, Nov 25, 2019 at 3:44 PM sebb <se...@gmail.com> wrote:
> > >
> > > On Mon, 25 Nov 2019 at 19:48, Sam Ruby <ru...@intertwingly.net> wrote:
> > >
> > > > On Mon, Nov 25, 2019 at 2:29 PM sebb <se...@gmail.com> wrote:
> > > > >
> > > > > On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net>
> > wrote:
> > > > >
> > > > > > On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
> > > > > > >
> > > > > > > I was thinking of using svn ls to create a listing file which
> > would
> > > > be
> > > > > > > cached locally.
> > > > > >
> > > > > > Unfortunately, some observations (numbers below are approximate):
> > > > > >
> > > > > > svn up on a populated iclas directory: one second
> > > > > >
> > > > > > svn ls on iclas: two seconds, but only returns depth one
> > > > > >
> > > > > > svn ls on iclas --depth infinity: one minute
> > > > > >
> > > > > > What's most concerning is not just the elapsed time, but that
> this
> > > > > > likely means that the call is expensive on the server, which may
> > > > > > impact other users.
> > > > > >
> > > > > >
> > > > > I see this as premature optimisation; we don't know whether svn
> list
> > is
> > > > > more expensive than svn update overall.
> > > > > There may be other reasons why list is slower. Nor do we know if
> the
> > > > > request will impact other users.
> > > > >
> > > > > Besides, if the code checks SVN info first, it will only need to
> > fetch
> > > > the
> > > > > updated listing when there has been a change.
> > > > > Those directories are not busy.
> > > > >
> > > > > Furthermore, every time a new test installation is set up, there is
> > > > > definitely a large load on the server and network.
> > > > > This network load in particular must be orders of magnitude greater
> > than
> > > > > for a listing.
> > > >
> > > > Sorry for not being clear. I would be very concerned if whimsy-vm4
> > > > were invoking svn ls --depth infinity every 10 minutes as the current
> > > > cron job does.
> > >
> > >
> > > That would not be the case.
> > >
> > > The job would use 'svn info' on the remote repo and only fetch the
> > listing
> > > if necessary.
> > >
> > > For the repos in question, changes are rare.
> > >
> > >
> > > > Before any such change is deployed, it would be wise
> > > > for us to check both with the infrastructure team and the subversion
> > > > team (Greg likely can help with both).
> > > >
> > > > I'm less concerned about the overhead on development machines, and
> > > > there I suspect that most users would be happy with a svn checkout
> > > > --depth empty.
> > > >
> > > >
> > > This would not allow testing of the functions that need to know the
> list
> > of
> > > file names.
> > >
> > >
> > > > - Sam Ruby
> > > >
> >
>
Re: Does Whimsy need to have a copy of Bills?
Posted by Greg Stein <gs...@gmail.com>.
One quick answer from earlier in the thread:
> > > > What's most concerning is not just the elapsed time, but that this
> > > > likely means that the call is expensive on the server, which may
> > > > impact other users.
> > > >
> > > >
> > > I see this as premature optimisation; we don't know whether svn list
is
> > > more expensive than svn update overall.
> > > There may be other reasons why list is slower. Nor do we know if the
> > > request will impact other users.
"svn list" must generate the listing of 12k+ files, recursively. That takes
some time to process and deliver over the network. I believe it is likely a
PROPFIND which introduces some overheads on both ends (XML construction on
server, parsing on client; sheer network size, too).
"svn up" generates a diff report. "get these 3 files", and that's easily
extracted from the difference between revision-working-copy and
revision-server (plus some other concerns).
So yes: an update is *way* faster, all around.
On Mon, Nov 25, 2019 at 5:15 PM Sam Ruby <ru...@intertwingly.net> wrote:
> Actually copy Greg this time.
>
> ---------- Forwarded message ---------
> From: Sam Ruby <ru...@intertwingly.net>
> Date: Mon, Nov 25, 2019, 4:15 PM
> Subject: Re: Does Whimsy need to have a copy of Bills?
> To: Whimsy dev <de...@whimsical.apache.org>
>
>
> adding Greg to email.
>
> Recap: a change is being proposed whereas whimsy will do the
> equivalent of the following command after every icla is processed:
>
> svn ls https://svn.apache.org/repos/private/documents/iclas --depth
> infinity
>
Do you really need to use depth=infinity? The directory name is likely
sufficient information. ?
depth=immediates (the default for svn ls) is going to be just a few seconds.
Currently this appears to take around forty to sixty elapsed seconds to
> process.
>
> Questions for Greg:
> 1) Does this proposed workload present an unreasonable load on the svn
> server?
>
This should be fine as long as you don't put a "-v" switch in there.
That'll take 10-15 minutes as it reconstructs all the files on the server
and measures their size.
The listing is just a single thread on the server, a fetch of the directory
names, and then assembly/delivery of that result. There really shouldn't be
any contention with other users, or heavy use of the CPU.
> 2) Are there any faster alternatives which get us a list of names but no
> data
>
So I experimented with a hack. I did a "full checkout" of the iclas
directory, but stopped it after a single file was checked out. This left a
partial checkout. Subversion will tell the server "I have $these. what am I
missing?" when you run "svn status -u". You'll get a listing of the 12k
missing files. Takes about 7 seconds or so.
Specifically:
$ # kill after reading/printing one line (the first file checked out)
$ svn co https://svn.apache.org/repos/private/documents/iclas | python -c
'import signal,os,sys ; print sys.stdin.readline() ;
os.killpg(os.getpgrp(), signal.SIGHUP)'
A iclas/jiwei-guo
svn: E200015: Caught signal
svn: E200042: Additional errors:
svn: E200015: Caught signal
Hangup
$ # get a recursive listing via status
$ time svn st -u iclas | wc -l
12818
real 0m6.782s
user 0m3.461s
sys 0m2.358s
The status output should be easy to parse (it is designed as a fixed-width
set of codes, then filename).
Even if you do a full/normal checkout, note that "svn status -u" may be
useful. Depending on whether you need the content, or just the names, you
may want to migrate to the status-based approach.
Oh! Just realized a better way, to avoid the hack/partial checkout. Even
better, just check out the "iclas" directory for the revision it was
created. It is an empty directory in that revision (a sibling directory
received a bunch of Member applications, but those won't be in this
checkout).
$ svn -r 9696 co https://svn.apache.org/repos/private/documents/iclas
Checked out revision 9696.
The "svn status" works the same against the above (empty) working copy.
Also at about 8 seconds.
So. In summary, use "svn status" against a HEAD checkout, or against r9696
for those who don't want the gigabytes of ICLA forms.
A similar technique can be used for any of the other Whimsy data
directories, of course. To find when a particular directory was created:
$ # run an "svn log" in reverse order, and limit/stop at the first log
entry.
$ svn log --stop-on-copy --limit 1 -r0:HEAD
https://svn.apache.org/repos/private/documents/iclas
------------------------------------------------------------------------
r9696 | jim | 2006-11-17 10:35:28 -0600 (Fri, 17 Nov 2006) | 3 lines
Start loading of scanned docs. Start with creating
the dirs and upload the member apps
------------------------------------------------------------------------
Note that sometimes a directory is created with content, in that revision.
The "iclas" directory just happened to be created empty. But I imagine most
directories will be much smaller at their creation, than they are today.
(iow, don't expect them to always be empty at creation)
Hope that helps,
-g
> - Sam Ruby
>
> On Mon, Nov 25, 2019 at 3:44 PM sebb <se...@gmail.com> wrote:
> >
> > On Mon, 25 Nov 2019 at 19:48, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > > On Mon, Nov 25, 2019 at 2:29 PM sebb <se...@gmail.com> wrote:
> > > >
> > > > On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net>
> wrote:
> > > >
> > > > > On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
> > > > > >
> > > > > > I was thinking of using svn ls to create a listing file which
> would
> > > be
> > > > > > cached locally.
> > > > >
> > > > > Unfortunately, some observations (numbers below are approximate):
> > > > >
> > > > > svn up on a populated iclas directory: one second
> > > > >
> > > > > svn ls on iclas: two seconds, but only returns depth one
> > > > >
> > > > > svn ls on iclas --depth infinity: one minute
> > > > >
> > > > > What's most concerning is not just the elapsed time, but that this
> > > > > likely means that the call is expensive on the server, which may
> > > > > impact other users.
> > > > >
> > > > >
> > > > I see this as premature optimisation; we don't know whether svn list
> is
> > > > more expensive than svn update overall.
> > > > There may be other reasons why list is slower. Nor do we know if the
> > > > request will impact other users.
> > > >
> > > > Besides, if the code checks SVN info first, it will only need to
> fetch
> > > the
> > > > updated listing when there has been a change.
> > > > Those directories are not busy.
> > > >
> > > > Furthermore, every time a new test installation is set up, there is
> > > > definitely a large load on the server and network.
> > > > This network load in particular must be orders of magnitude greater
> than
> > > > for a listing.
> > >
> > > Sorry for not being clear. I would be very concerned if whimsy-vm4
> > > were invoking svn ls --depth infinity every 10 minutes as the current
> > > cron job does.
> >
> >
> > That would not be the case.
> >
> > The job would use 'svn info' on the remote repo and only fetch the
> listing
> > if necessary.
> >
> > For the repos in question, changes are rare.
> >
> >
> > > Before any such change is deployed, it would be wise
> > > for us to check both with the infrastructure team and the subversion
> > > team (Greg likely can help with both).
> > >
> > > I'm less concerned about the overhead on development machines, and
> > > there I suspect that most users would be happy with a svn checkout
> > > --depth empty.
> > >
> > >
> > This would not allow testing of the functions that need to know the list
> of
> > file names.
> >
> >
> > > - Sam Ruby
> > >
>
Fwd: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
Actually copy Greg this time.
---------- Forwarded message ---------
From: Sam Ruby <ru...@intertwingly.net>
Date: Mon, Nov 25, 2019, 4:15 PM
Subject: Re: Does Whimsy need to have a copy of Bills?
To: Whimsy dev <de...@whimsical.apache.org>
adding Greg to email.
Recap: a change is being proposed whereas whimsy will do the
equivalent of the following command after every icla is processed:
svn ls https://svn.apache.org/repos/private/documents/iclas --depth infinity
Currently this appears to take around forty to sixty elapsed seconds to
process.
Questions for Greg:
1) Does this proposed workload present an unreasonable load on the svn
server?
2) Are there any faster alternatives which get us a list of names but no
data
- Sam Ruby
On Mon, Nov 25, 2019 at 3:44 PM sebb <se...@gmail.com> wrote:
>
> On Mon, 25 Nov 2019 at 19:48, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > On Mon, Nov 25, 2019 at 2:29 PM sebb <se...@gmail.com> wrote:
> > >
> > > On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net> wrote:
> > >
> > > > On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
> > > > >
> > > > > I was thinking of using svn ls to create a listing file which
would
> > be
> > > > > cached locally.
> > > >
> > > > Unfortunately, some observations (numbers below are approximate):
> > > >
> > > > svn up on a populated iclas directory: one second
> > > >
> > > > svn ls on iclas: two seconds, but only returns depth one
> > > >
> > > > svn ls on iclas --depth infinity: one minute
> > > >
> > > > What's most concerning is not just the elapsed time, but that this
> > > > likely means that the call is expensive on the server, which may
> > > > impact other users.
> > > >
> > > >
> > > I see this as premature optimisation; we don't know whether svn list
is
> > > more expensive than svn update overall.
> > > There may be other reasons why list is slower. Nor do we know if the
> > > request will impact other users.
> > >
> > > Besides, if the code checks SVN info first, it will only need to fetch
> > the
> > > updated listing when there has been a change.
> > > Those directories are not busy.
> > >
> > > Furthermore, every time a new test installation is set up, there is
> > > definitely a large load on the server and network.
> > > This network load in particular must be orders of magnitude greater
than
> > > for a listing.
> >
> > Sorry for not being clear. I would be very concerned if whimsy-vm4
> > were invoking svn ls --depth infinity every 10 minutes as the current
> > cron job does.
>
>
> That would not be the case.
>
> The job would use 'svn info' on the remote repo and only fetch the listing
> if necessary.
>
> For the repos in question, changes are rare.
>
>
> > Before any such change is deployed, it would be wise
> > for us to check both with the infrastructure team and the subversion
> > team (Greg likely can help with both).
> >
> > I'm less concerned about the overhead on development machines, and
> > there I suspect that most users would be happy with a svn checkout
> > --depth empty.
> >
> >
> This would not allow testing of the functions that need to know the list
of
> file names.
>
>
> > - Sam Ruby
> >
Re: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
adding Greg to email.
Recap: a change is being proposed whereas whimsy will do the
equivalent of the following command after every icla is processed:
svn ls https://svn.apache.org/repos/private/documents/iclas --depth infinity
Currently this appears to take around forty to sixty elapsed seconds to process.
Questions for Greg:
1) Does this proposed workload present an unreasonable load on the svn server?
2) Are there any faster alternatives which get us a list of names but no data
- Sam Ruby
On Mon, Nov 25, 2019 at 3:44 PM sebb <se...@gmail.com> wrote:
>
> On Mon, 25 Nov 2019 at 19:48, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > On Mon, Nov 25, 2019 at 2:29 PM sebb <se...@gmail.com> wrote:
> > >
> > > On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net> wrote:
> > >
> > > > On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
> > > > >
> > > > > I was thinking of using svn ls to create a listing file which would
> > be
> > > > > cached locally.
> > > >
> > > > Unfortunately, some observations (numbers below are approximate):
> > > >
> > > > svn up on a populated iclas directory: one second
> > > >
> > > > svn ls on iclas: two seconds, but only returns depth one
> > > >
> > > > svn ls on iclas --depth infinity: one minute
> > > >
> > > > What's most concerning is not just the elapsed time, but that this
> > > > likely means that the call is expensive on the server, which may
> > > > impact other users.
> > > >
> > > >
> > > I see this as premature optimisation; we don't know whether svn list is
> > > more expensive than svn update overall.
> > > There may be other reasons why list is slower. Nor do we know if the
> > > request will impact other users.
> > >
> > > Besides, if the code checks SVN info first, it will only need to fetch
> > the
> > > updated listing when there has been a change.
> > > Those directories are not busy.
> > >
> > > Furthermore, every time a new test installation is set up, there is
> > > definitely a large load on the server and network.
> > > This network load in particular must be orders of magnitude greater than
> > > for a listing.
> >
> > Sorry for not being clear. I would be very concerned if whimsy-vm4
> > were invoking svn ls --depth infinity every 10 minutes as the current
> > cron job does.
>
>
> That would not be the case.
>
> The job would use 'svn info' on the remote repo and only fetch the listing
> if necessary.
>
> For the repos in question, changes are rare.
>
>
> > Before any such change is deployed, it would be wise
> > for us to check both with the infrastructure team and the subversion
> > team (Greg likely can help with both).
> >
> > I'm less concerned about the overhead on development machines, and
> > there I suspect that most users would be happy with a svn checkout
> > --depth empty.
> >
> >
> This would not allow testing of the functions that need to know the list of
> file names.
>
>
> > - Sam Ruby
> >
Re: Does Whimsy need to have a copy of Bills?
Posted by sebb <se...@gmail.com>.
On Mon, 25 Nov 2019 at 19:48, Sam Ruby <ru...@intertwingly.net> wrote:
> On Mon, Nov 25, 2019 at 2:29 PM sebb <se...@gmail.com> wrote:
> >
> > On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > > On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
> > > >
> > > > I was thinking of using svn ls to create a listing file which would
> be
> > > > cached locally.
> > >
> > > Unfortunately, some observations (numbers below are approximate):
> > >
> > > svn up on a populated iclas directory: one second
> > >
> > > svn ls on iclas: two seconds, but only returns depth one
> > >
> > > svn ls on iclas --depth infinity: one minute
> > >
> > > What's most concerning is not just the elapsed time, but that this
> > > likely means that the call is expensive on the server, which may
> > > impact other users.
> > >
> > >
> > I see this as premature optimisation; we don't know whether svn list is
> > more expensive than svn update overall.
> > There may be other reasons why list is slower. Nor do we know if the
> > request will impact other users.
> >
> > Besides, if the code checks SVN info first, it will only need to fetch
> the
> > updated listing when there has been a change.
> > Those directories are not busy.
> >
> > Furthermore, every time a new test installation is set up, there is
> > definitely a large load on the server and network.
> > This network load in particular must be orders of magnitude greater than
> > for a listing.
>
> Sorry for not being clear. I would be very concerned if whimsy-vm4
> were invoking svn ls --depth infinity every 10 minutes as the current
> cron job does.
That would not be the case.
The job would use 'svn info' on the remote repo and only fetch the listing
if necessary.
For the repos in question, changes are rare.
> Before any such change is deployed, it would be wise
> for us to check both with the infrastructure team and the subversion
> team (Greg likely can help with both).
>
> I'm less concerned about the overhead on development machines, and
> there I suspect that most users would be happy with a svn checkout
> --depth empty.
>
>
This would not allow testing of the functions that need to know the list of
file names.
> - Sam Ruby
>
Re: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Mon, Nov 25, 2019 at 2:29 PM sebb <se...@gmail.com> wrote:
>
> On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
> > >
> > > I was thinking of using svn ls to create a listing file which would be
> > > cached locally.
> >
> > Unfortunately, some observations (numbers below are approximate):
> >
> > svn up on a populated iclas directory: one second
> >
> > svn ls on iclas: two seconds, but only returns depth one
> >
> > svn ls on iclas --depth infinity: one minute
> >
> > What's most concerning is not just the elapsed time, but that this
> > likely means that the call is expensive on the server, which may
> > impact other users.
> >
> >
> I see this as premature optimisation; we don't know whether svn list is
> more expensive than svn update overall.
> There may be other reasons why list is slower. Nor do we know if the
> request will impact other users.
>
> Besides, if the code checks SVN info first, it will only need to fetch the
> updated listing when there has been a change.
> Those directories are not busy.
>
> Furthermore, every time a new test installation is set up, there is
> definitely a large load on the server and network.
> This network load in particular must be orders of magnitude greater than
> for a listing.
Sorry for not being clear. I would be very concerned if whimsy-vm4
were invoking svn ls --depth infinity every 10 minutes as the current
cron job does. Before any such change is deployed, it would be wise
for us to check both with the infrastructure team and the subversion
team (Greg likely can help with both).
I'm less concerned about the overhead on development machines, and
there I suspect that most users would be happy with a svn checkout
--depth empty.
- Sam Ruby
Re: Does Whimsy need to have a copy of Bills?
Posted by sebb <se...@gmail.com>.
On Mon, 25 Nov 2019 at 17:59, Sam Ruby <ru...@intertwingly.net> wrote:
> On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
> >
> > I was thinking of using svn ls to create a listing file which would be
> > cached locally.
>
> Unfortunately, some observations (numbers below are approximate):
>
> svn up on a populated iclas directory: one second
>
> svn ls on iclas: two seconds, but only returns depth one
>
> svn ls on iclas --depth infinity: one minute
>
> What's most concerning is not just the elapsed time, but that this
> likely means that the call is expensive on the server, which may
> impact other users.
>
>
I see this as premature optimisation; we don't know whether svn list is
more expensive than svn update overall.
There may be other reasons why list is slower. Nor do we know if the
request will impact other users.
Besides, if the code checks SVN info first, it will only need to fetch the
updated listing when there has been a change.
Those directories are not busy.
Furthermore, every time a new test installation is set up, there is
definitely a large load on the server and network.
This network load in particular must be orders of magnitude greater than
for a listing.
> - Sam Ruby
>
Re: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Sun, Nov 24, 2019 at 1:17 PM sebb <se...@gmail.com> wrote:
>
> I was thinking of using svn ls to create a listing file which would be
> cached locally.
Unfortunately, some observations (numbers below are approximate):
svn up on a populated iclas directory: one second
svn ls on iclas: two seconds, but only returns depth one
svn ls on iclas --depth infinity: one minute
What's most concerning is not just the elapsed time, but that this
likely means that the call is expensive on the server, which may
impact other users.
- Sam Ruby
Re: Does Whimsy need to have a copy of Bills?
Posted by sebb <se...@gmail.com>.
On Sun, 24 Nov 2019 at 15:30, Sam Ruby <ru...@intertwingly.net> wrote:
> On Sun, Nov 24, 2019 at 7:22 AM sebb <se...@gmail.com> wrote:
> >
> > On Sun, 24 Nov 2019 at 01:53, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > > On Sat, Nov 23, 2019 at 8:30 PM Dave Fisher <wa...@comcast.net>
> wrote:
> > > >
> > > > It would be great to have a mapping between whimsy sections and svn
> > > paths, and allow whimsy developers to choose which svn paths are used
> for
> > > their current efforts.
> > >
> > > https://github.com/apache/whimsy/blob/master/repository.yml
> > > https://whimsy.apache.org/test/dataflow
> > >
> > > The latter is a work in progress. Any efforts to fill in the blanks
> > > would be appreciated
> > >
> > > > I’ve felt uncomfortable with all the information I have to download
> to
> > > work on whimsy.
> > >
> > > I'm envisioning adding a configuration option where one can list which
> > > of these paths to either include or exclude (at your choice) from
> > > updates.
> > >
> > > My thoughts are that things that are small don't cause much concern.
> > > Here is the current sizes of the svn checkouts, in megabytes:
> > >
> > > 19331 iclas
> > > 1313 cclas
> > > 823 grants
> > > 699 Bills
> > > 673 member_apps
> > >
> >
> > I don't think any of the above are actually needed, except to check for
> > existence.
>
> So these could be replaced by calls to svn info? We would just have
> to handle svn being down.
>
>
I was thinking of using svn ls to create a listing file which would be
cached locally.
svn info could be used to determine whether the local cache is up to date.
When creating a new file, Whimsy could use:
svnmucc -r 0 put <local> <remote>
This will fail if the remote is not at version 0, i.e. if it exists.
Similarly for files that are to be updated, except use what Whimsy thinks
is the current version number.
> - Sam Ruby
>
> > > 59 foundation_board
> > > 53 Meetings
> > > 51 minutes
> > > 10 site-img
> > > 6 steve
> > > 6 comdev-foundation
> > > 5 officers
> > > 4 foundation
> > > 3 board
> > > 3 apmail_bin
> > > 3 apachecon
> > > 2 officers_historic
> > > 2 incubator-content
> > > 2 brandlist
> > > 1 templates
> > > 1 site-root
> > > 1 site-board
> > > 1 qmail_control
> > > 1 personnel-duties
> > > 1 officers_surveys
> > > 1 ldap-map
> > > 1 incubator-podlings
> > > 1 foundation_mentors
> > > 1 comdevtalks
> > > 1 attic-xdocs
> > >
> > > > Sent from my iPhone
> > >
> > > - Sam Ruby
> > >
> > > > > On Nov 23, 2019, at 5:07 PM, Sam Ruby <ru...@intertwingly.net>
> wrote:
> > > > >
> > > > > On Sat, Nov 23, 2019 at 11:18 AM sebb <se...@gmail.com> wrote:
> > > > >>
> > > > >> AFAICT Whimsy does not use its local checkout of Bills, so the
> > > directory
> > > > >> could be dropped.
> > > > >
> > > > > Agreed.
> > > > >
> > > > > - Sam Ruby
> > > >
> > >
>
Re: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Sun, Nov 24, 2019 at 7:22 AM sebb <se...@gmail.com> wrote:
>
> On Sun, 24 Nov 2019 at 01:53, Sam Ruby <ru...@intertwingly.net> wrote:
>
> > On Sat, Nov 23, 2019 at 8:30 PM Dave Fisher <wa...@comcast.net> wrote:
> > >
> > > It would be great to have a mapping between whimsy sections and svn
> > paths, and allow whimsy developers to choose which svn paths are used for
> > their current efforts.
> >
> > https://github.com/apache/whimsy/blob/master/repository.yml
> > https://whimsy.apache.org/test/dataflow
> >
> > The latter is a work in progress. Any efforts to fill in the blanks
> > would be appreciated
> >
> > > I’ve felt uncomfortable with all the information I have to download to
> > work on whimsy.
> >
> > I'm envisioning adding a configuration option where one can list which
> > of these paths to either include or exclude (at your choice) from
> > updates.
> >
> > My thoughts are that things that are small don't cause much concern.
> > Here is the current sizes of the svn checkouts, in megabytes:
> >
> > 19331 iclas
> > 1313 cclas
> > 823 grants
> > 699 Bills
> > 673 member_apps
> >
>
> I don't think any of the above are actually needed, except to check for
> existence.
So these could be replaced by calls to svn info? We would just have
to handle svn being down.
- Sam Ruby
> > 59 foundation_board
> > 53 Meetings
> > 51 minutes
> > 10 site-img
> > 6 steve
> > 6 comdev-foundation
> > 5 officers
> > 4 foundation
> > 3 board
> > 3 apmail_bin
> > 3 apachecon
> > 2 officers_historic
> > 2 incubator-content
> > 2 brandlist
> > 1 templates
> > 1 site-root
> > 1 site-board
> > 1 qmail_control
> > 1 personnel-duties
> > 1 officers_surveys
> > 1 ldap-map
> > 1 incubator-podlings
> > 1 foundation_mentors
> > 1 comdevtalks
> > 1 attic-xdocs
> >
> > > Sent from my iPhone
> >
> > - Sam Ruby
> >
> > > > On Nov 23, 2019, at 5:07 PM, Sam Ruby <ru...@intertwingly.net> wrote:
> > > >
> > > > On Sat, Nov 23, 2019 at 11:18 AM sebb <se...@gmail.com> wrote:
> > > >>
> > > >> AFAICT Whimsy does not use its local checkout of Bills, so the
> > directory
> > > >> could be dropped.
> > > >
> > > > Agreed.
> > > >
> > > > - Sam Ruby
> > >
> >
Re: Does Whimsy need to have a copy of Bills?
Posted by sebb <se...@gmail.com>.
On Sun, 24 Nov 2019 at 01:53, Sam Ruby <ru...@intertwingly.net> wrote:
> On Sat, Nov 23, 2019 at 8:30 PM Dave Fisher <wa...@comcast.net> wrote:
> >
> > It would be great to have a mapping between whimsy sections and svn
> paths, and allow whimsy developers to choose which svn paths are used for
> their current efforts.
>
> https://github.com/apache/whimsy/blob/master/repository.yml
> https://whimsy.apache.org/test/dataflow
>
> The latter is a work in progress. Any efforts to fill in the blanks
> would be appreciated
>
> > I’ve felt uncomfortable with all the information I have to download to
> work on whimsy.
>
> I'm envisioning adding a configuration option where one can list which
> of these paths to either include or exclude (at your choice) from
> updates.
>
> My thoughts are that things that are small don't cause much concern.
> Here is the current sizes of the svn checkouts, in megabytes:
>
> 19331 iclas
> 1313 cclas
> 823 grants
> 699 Bills
> 673 member_apps
>
I don't think any of the above are actually needed, except to check for
existence.
> 59 foundation_board
> 53 Meetings
> 51 minutes
> 10 site-img
> 6 steve
> 6 comdev-foundation
> 5 officers
> 4 foundation
> 3 board
> 3 apmail_bin
> 3 apachecon
> 2 officers_historic
> 2 incubator-content
> 2 brandlist
> 1 templates
> 1 site-root
> 1 site-board
> 1 qmail_control
> 1 personnel-duties
> 1 officers_surveys
> 1 ldap-map
> 1 incubator-podlings
> 1 foundation_mentors
> 1 comdevtalks
> 1 attic-xdocs
>
> > Sent from my iPhone
>
> - Sam Ruby
>
> > > On Nov 23, 2019, at 5:07 PM, Sam Ruby <ru...@intertwingly.net> wrote:
> > >
> > > On Sat, Nov 23, 2019 at 11:18 AM sebb <se...@gmail.com> wrote:
> > >>
> > >> AFAICT Whimsy does not use its local checkout of Bills, so the
> directory
> > >> could be dropped.
> > >
> > > Agreed.
> > >
> > > - Sam Ruby
> >
>
Re: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Sat, Nov 23, 2019 at 8:30 PM Dave Fisher <wa...@comcast.net> wrote:
>
> It would be great to have a mapping between whimsy sections and svn paths, and allow whimsy developers to choose which svn paths are used for their current efforts.
https://github.com/apache/whimsy/blob/master/repository.yml
https://whimsy.apache.org/test/dataflow
The latter is a work in progress. Any efforts to fill in the blanks
would be appreciated
> I’ve felt uncomfortable with all the information I have to download to work on whimsy.
I'm envisioning adding a configuration option where one can list which
of these paths to either include or exclude (at your choice) from
updates.
My thoughts are that things that are small don't cause much concern.
Here is the current sizes of the svn checkouts, in megabytes:
19331 iclas
1313 cclas
823 grants
699 Bills
673 member_apps
59 foundation_board
53 Meetings
51 minutes
10 site-img
6 steve
6 comdev-foundation
5 officers
4 foundation
3 board
3 apmail_bin
3 apachecon
2 officers_historic
2 incubator-content
2 brandlist
1 templates
1 site-root
1 site-board
1 qmail_control
1 personnel-duties
1 officers_surveys
1 ldap-map
1 incubator-podlings
1 foundation_mentors
1 comdevtalks
1 attic-xdocs
> Sent from my iPhone
- Sam Ruby
> > On Nov 23, 2019, at 5:07 PM, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > On Sat, Nov 23, 2019 at 11:18 AM sebb <se...@gmail.com> wrote:
> >>
> >> AFAICT Whimsy does not use its local checkout of Bills, so the directory
> >> could be dropped.
> >
> > Agreed.
> >
> > - Sam Ruby
>
Re: Does Whimsy need to have a copy of Bills?
Posted by Dave Fisher <wa...@comcast.net>.
It would be great to have a mapping between whimsy sections and svn paths, and allow whimsy developers to choose which svn paths are used for their current efforts.
I’ve felt uncomfortable with all the information I have to download to work on whimsy.
Sent from my iPhone
> On Nov 23, 2019, at 5:07 PM, Sam Ruby <ru...@intertwingly.net> wrote:
>
> On Sat, Nov 23, 2019 at 11:18 AM sebb <se...@gmail.com> wrote:
>>
>> AFAICT Whimsy does not use its local checkout of Bills, so the directory
>> could be dropped.
>
> Agreed.
>
> - Sam Ruby
Re: Does Whimsy need to have a copy of Bills?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Sat, Nov 23, 2019 at 11:18 AM sebb <se...@gmail.com> wrote:
>
> AFAICT Whimsy does not use its local checkout of Bills, so the directory
> could be dropped.
Agreed.
- Sam Ruby