You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whimsical.apache.org by sebb <se...@gmail.com> on 2016/03/02 14:09:41 UTC
Access to single SVN items - implement caching?
Some of the code needs access to a single file in SVN, e.g.
https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings.xml
AFAIK there are no other items needed in the same part of SVN.
Yet one has to checkout the parent directory and keep it updated.
This is a nuisance for testing purposes.
For SVN files that don't need authentication, it is easy enough to
fetch the file using HTTP(S) and cache it locally. For subsequent
requests, the cache date can be used in an If-Modified request.
The last checked date can also be saved to avoid checking the file too often.
This technique has been used in reporter.a.o for checking the chair
index in foundation/index.mdtext:
https://svn.apache.org/repos/asf/comdev/reporter.apache.org/trunk/scripts/parsepmcs.py
Might be worth cosidering re-implementing the python caching code used there.
Re: Access to single SVN items - implement caching?
Posted by sebb <se...@gmail.com>.
On 4 March 2016 at 15:40, Sam Ruby <ru...@intertwingly.net> wrote:
> On Fri, Mar 4, 2016 at 10:03 AM, sebb <se...@gmail.com> wrote:
>> On 4 March 2016 at 00:19, Sam Ruby <ru...@intertwingly.net> wrote:
>>> On Thu, Mar 3, 2016 at 4:35 PM, sebb <se...@gmail.com> wrote:
>>>> On 3 March 2016 at 21:12, Sam Ruby <ru...@intertwingly.net> wrote:
>>>>> On Thu, Mar 3, 2016 at 3:50 PM, sebb <se...@gmail.com> wrote:
>>>>>> On 3 March 2016 at 13:28, Sam Ruby <ru...@intertwingly.net> wrote:
>>>>>>> On Wed, Mar 2, 2016 at 9:22 AM, sebb <se...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Better would be to have a function to report which entries are
>>>>>>>> missing, so the user can decide which they want to download based on
>>>>>>>> what they are testing.
>>>>>>>>
>>>>>>>> Maybe also with a function to report which are out of date.
>>>>>>>
>>>>>>> I've roughed in such a tool. You will need to do a "bundle update" or
>>>>>>> otherwise get the latest Wunderbar (1.0.21) to test it out locally.
>>>>>>> Then run:
>>>>>>>
>>>>>>> ruby www/status/svn.cgi
>>>>>>>
>>>>>>> And then visit http://localhost:4567/
>>>>>>
>>>>>> That's very cool.
>>>>>>
>>>>>> I'd like to add the local path details if not in read-only mode.
>>>>>> Would that be OK?
>>>>>
>>>>> Local path details are already there :-)
>>>>>
>>>>> Go to https://whimsy.apache.org/status/svn and hover over any text in
>>>>> the first column.
>>>>
>>>> I see, that's useful.
>>>>
>>>> However one cannot copy the hover text nor search it except by looking
>>>> at the source.
>>>
>>> OK, so how about a column that only shows if the window is wide enough?
>>>
>>> https://github.com/apache/whimsy/commit/c26dbfdd4b0a258b3fdbdb7f122d0da4e6b606c6
>>
>> Great, though maybe it should be disabled for the read-only case i.e.
>> on the live system.
>
> I don't see the harm.
No harm, but it's not particularly useful in that case.
So it's better not to clutter the screen.
>> Maybe it could depend on whether the script was run directly or via
>> the webserver?
>>
>> If run under a webserver, one probably does not want the buttons either.
>
> I'm currently running it under a webserver with suexec enabled, and
> the buttons work for me. That being said, if/when you try out
> https://whimsy.apache.org/test/vhost-generator and it gets to the
> point where it works for you, I plan to switch to that. The secretary
> workbench is the only remaining application that updates files in the
> svn working directory (something that has proven to be problematic),
> and its eventual replacement (the secmail tool) will no longer do
> that. So I won't need suexec any more.
>
> In any case, you don't need my permission to make updates.
OK, thanks.
But it's more a question of ensuring that I'm not overlooking other
aspects of the app and committing unnecessary/conflicting changes.
>>>>> I did it this way so that super-wide screens aren't required.
>>>>>
>>>>> - Sam Ruby
>>>
>>> - Sam Ruby
>
> - Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Fri, Mar 4, 2016 at 10:03 AM, sebb <se...@gmail.com> wrote:
> On 4 March 2016 at 00:19, Sam Ruby <ru...@intertwingly.net> wrote:
>> On Thu, Mar 3, 2016 at 4:35 PM, sebb <se...@gmail.com> wrote:
>>> On 3 March 2016 at 21:12, Sam Ruby <ru...@intertwingly.net> wrote:
>>>> On Thu, Mar 3, 2016 at 3:50 PM, sebb <se...@gmail.com> wrote:
>>>>> On 3 March 2016 at 13:28, Sam Ruby <ru...@intertwingly.net> wrote:
>>>>>> On Wed, Mar 2, 2016 at 9:22 AM, sebb <se...@gmail.com> wrote:
>>>>>>>
>>>>>>> Better would be to have a function to report which entries are
>>>>>>> missing, so the user can decide which they want to download based on
>>>>>>> what they are testing.
>>>>>>>
>>>>>>> Maybe also with a function to report which are out of date.
>>>>>>
>>>>>> I've roughed in such a tool. You will need to do a "bundle update" or
>>>>>> otherwise get the latest Wunderbar (1.0.21) to test it out locally.
>>>>>> Then run:
>>>>>>
>>>>>> ruby www/status/svn.cgi
>>>>>>
>>>>>> And then visit http://localhost:4567/
>>>>>
>>>>> That's very cool.
>>>>>
>>>>> I'd like to add the local path details if not in read-only mode.
>>>>> Would that be OK?
>>>>
>>>> Local path details are already there :-)
>>>>
>>>> Go to https://whimsy.apache.org/status/svn and hover over any text in
>>>> the first column.
>>>
>>> I see, that's useful.
>>>
>>> However one cannot copy the hover text nor search it except by looking
>>> at the source.
>>
>> OK, so how about a column that only shows if the window is wide enough?
>>
>> https://github.com/apache/whimsy/commit/c26dbfdd4b0a258b3fdbdb7f122d0da4e6b606c6
>
> Great, though maybe it should be disabled for the read-only case i.e.
> on the live system.
I don't see the harm.
> Maybe it could depend on whether the script was run directly or via
> the webserver?
>
> If run under a webserver, one probably does not want the buttons either.
I'm currently running it under a webserver with suexec enabled, and
the buttons work for me. That being said, if/when you try out
https://whimsy.apache.org/test/vhost-generator and it gets to the
point where it works for you, I plan to switch to that. The secretary
workbench is the only remaining application that updates files in the
svn working directory (something that has proven to be problematic),
and its eventual replacement (the secmail tool) will no longer do
that. So I won't need suexec any more.
In any case, you don't need my permission to make updates.
>>>> I did it this way so that super-wide screens aren't required.
>>>>
>>>> - Sam Ruby
>>
>> - Sam Ruby
- Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by sebb <se...@gmail.com>.
On 4 March 2016 at 00:19, Sam Ruby <ru...@intertwingly.net> wrote:
> On Thu, Mar 3, 2016 at 4:35 PM, sebb <se...@gmail.com> wrote:
>> On 3 March 2016 at 21:12, Sam Ruby <ru...@intertwingly.net> wrote:
>>> On Thu, Mar 3, 2016 at 3:50 PM, sebb <se...@gmail.com> wrote:
>>>> On 3 March 2016 at 13:28, Sam Ruby <ru...@intertwingly.net> wrote:
>>>>> On Wed, Mar 2, 2016 at 9:22 AM, sebb <se...@gmail.com> wrote:
>>>>>>
>>>>>> Better would be to have a function to report which entries are
>>>>>> missing, so the user can decide which they want to download based on
>>>>>> what they are testing.
>>>>>>
>>>>>> Maybe also with a function to report which are out of date.
>>>>>
>>>>> I've roughed in such a tool. You will need to do a "bundle update" or
>>>>> otherwise get the latest Wunderbar (1.0.21) to test it out locally.
>>>>> Then run:
>>>>>
>>>>> ruby www/status/svn.cgi
>>>>>
>>>>> And then visit http://localhost:4567/
>>>>
>>>> That's very cool.
>>>>
>>>> I'd like to add the local path details if not in read-only mode.
>>>> Would that be OK?
>>>
>>> Local path details are already there :-)
>>>
>>> Go to https://whimsy.apache.org/status/svn and hover over any text in
>>> the first column.
>>
>> I see, that's useful.
>>
>> However one cannot copy the hover text nor search it except by looking
>> at the source.
>
> OK, so how about a column that only shows if the window is wide enough?
>
> https://github.com/apache/whimsy/commit/c26dbfdd4b0a258b3fdbdb7f122d0da4e6b606c6
Great, though maybe it should be disabled for the read-only case i.e.
on the live system.
Maybe it could depend on whether the script was run directly or via
the webserver?
If run under a webserver, one probably does not want the buttons either.
>>> I did it this way so that super-wide screens aren't required.
>>>
>>> - Sam Ruby
>
> - Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Thu, Mar 3, 2016 at 4:35 PM, sebb <se...@gmail.com> wrote:
> On 3 March 2016 at 21:12, Sam Ruby <ru...@intertwingly.net> wrote:
>> On Thu, Mar 3, 2016 at 3:50 PM, sebb <se...@gmail.com> wrote:
>>> On 3 March 2016 at 13:28, Sam Ruby <ru...@intertwingly.net> wrote:
>>>> On Wed, Mar 2, 2016 at 9:22 AM, sebb <se...@gmail.com> wrote:
>>>>>
>>>>> Better would be to have a function to report which entries are
>>>>> missing, so the user can decide which they want to download based on
>>>>> what they are testing.
>>>>>
>>>>> Maybe also with a function to report which are out of date.
>>>>
>>>> I've roughed in such a tool. You will need to do a "bundle update" or
>>>> otherwise get the latest Wunderbar (1.0.21) to test it out locally.
>>>> Then run:
>>>>
>>>> ruby www/status/svn.cgi
>>>>
>>>> And then visit http://localhost:4567/
>>>
>>> That's very cool.
>>>
>>> I'd like to add the local path details if not in read-only mode.
>>> Would that be OK?
>>
>> Local path details are already there :-)
>>
>> Go to https://whimsy.apache.org/status/svn and hover over any text in
>> the first column.
>
> I see, that's useful.
>
> However one cannot copy the hover text nor search it except by looking
> at the source.
OK, so how about a column that only shows if the window is wide enough?
https://github.com/apache/whimsy/commit/c26dbfdd4b0a258b3fdbdb7f122d0da4e6b606c6
>> I did it this way so that super-wide screens aren't required.
>>
>> - Sam Ruby
- Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by sebb <se...@gmail.com>.
On 3 March 2016 at 21:12, Sam Ruby <ru...@intertwingly.net> wrote:
> On Thu, Mar 3, 2016 at 3:50 PM, sebb <se...@gmail.com> wrote:
>> On 3 March 2016 at 13:28, Sam Ruby <ru...@intertwingly.net> wrote:
>>> On Wed, Mar 2, 2016 at 9:22 AM, sebb <se...@gmail.com> wrote:
>>>>
>>>> Better would be to have a function to report which entries are
>>>> missing, so the user can decide which they want to download based on
>>>> what they are testing.
>>>>
>>>> Maybe also with a function to report which are out of date.
>>>
>>> I've roughed in such a tool. You will need to do a "bundle update" or
>>> otherwise get the latest Wunderbar (1.0.21) to test it out locally.
>>> Then run:
>>>
>>> ruby www/status/svn.cgi
>>>
>>> And then visit http://localhost:4567/
>>
>> That's very cool.
>>
>> I'd like to add the local path details if not in read-only mode.
>> Would that be OK?
>
> Local path details are already there :-)
>
> Go to https://whimsy.apache.org/status/svn and hover over any text in
> the first column.
I see, that's useful.
However one cannot copy the hover text nor search it except by looking
at the source.
> I did it this way so that super-wide screens aren't required.
>
> - Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Thu, Mar 3, 2016 at 3:50 PM, sebb <se...@gmail.com> wrote:
> On 3 March 2016 at 13:28, Sam Ruby <ru...@intertwingly.net> wrote:
>> On Wed, Mar 2, 2016 at 9:22 AM, sebb <se...@gmail.com> wrote:
>>>
>>> Better would be to have a function to report which entries are
>>> missing, so the user can decide which they want to download based on
>>> what they are testing.
>>>
>>> Maybe also with a function to report which are out of date.
>>
>> I've roughed in such a tool. You will need to do a "bundle update" or
>> otherwise get the latest Wunderbar (1.0.21) to test it out locally.
>> Then run:
>>
>> ruby www/status/svn.cgi
>>
>> And then visit http://localhost:4567/
>
> That's very cool.
>
> I'd like to add the local path details if not in read-only mode.
> Would that be OK?
Local path details are already there :-)
Go to https://whimsy.apache.org/status/svn and hover over any text in
the first column.
I did it this way so that super-wide screens aren't required.
- Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by sebb <se...@gmail.com>.
On 3 March 2016 at 13:28, Sam Ruby <ru...@intertwingly.net> wrote:
> On Wed, Mar 2, 2016 at 9:22 AM, sebb <se...@gmail.com> wrote:
>> On 2 March 2016 at 13:52, Sam Ruby <ru...@intertwingly.net> wrote:
>>> On Wed, Mar 2, 2016 at 8:09 AM, sebb <se...@gmail.com> wrote:
>>>> Some of the code needs access to a single file in SVN, e.g.
>>>>
>>>> https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings.xml
>>>>
>>>> AFAIK there are no other items needed in the same part of SVN.
>>>>
>>>> Yet one has to checkout the parent directory and keep it updated.
>>>>
>>>> This is a nuisance for testing purposes.
>>>>
>>>> For SVN files that don't need authentication, it is easy enough to
>>>> fetch the file using HTTP(S) and cache it locally. For subsequent
>>>> requests, the cache date can be used in an If-Modified request.
>>>> The last checked date can also be saved to avoid checking the file too often.
>>>>
>>>> This technique has been used in reporter.a.o for checking the chair
>>>> index in foundation/index.mdtext:
>>>>
>>>> https://svn.apache.org/repos/asf/comdev/reporter.apache.org/trunk/scripts/parsepmcs.py
>>>>
>>>> Might be worth cosidering re-implementing the python caching code used there.
>>>
>>> As very few of the resources accessed by whimsy are public, that would
>>> only solve a small part of the problem.
>>
>> Which in itself is an issue - should we be encouraging downloading all
>> these resources for testing purposes?
>>
>> In particular, it seems wrong to have to download all the ICLAs.
>
> Ideally, applications would only make optional use of such
> repositories - and would check to see if ASF::SVN returned nil or not.
>
>>> Given that the resources needed are declared in
>>> https://github.com/apache/whimsy/blob/master/repository.yml, perhaps a
>>> more complete solution would be to use the 'svn:update' task defined
>>> in https://github.com/apache/whimsy/blob/master/Rakefile#L66 ?
>>>
>>> At the moment, that task is assuming that you do *NOT* override :svn:
>>> in your .whimsy config file. Perhaps it should handle that case?
>>
>> Better would be to have a function to report which entries are
>> missing, so the user can decide which they want to download based on
>> what they are testing.
>>
>> Maybe also with a function to report which are out of date.
>
> I've roughed in such a tool. You will need to do a "bundle update" or
> otherwise get the latest Wunderbar (1.0.21) to test it out locally.
> Then run:
>
> ruby www/status/svn.cgi
>
> And then visit http://localhost:4567/
>
That's very cool.
I'd like to add the local path details if not in read-only mode.
Would that be OK?
> Checkout is not yet implemented (it would need to prompt you for a destination).
>
> This tool is deployed on whimsy, where it runs in read-only mode:
>
> https://whimsy.apache.org/status/svn
>
>> It might be useful if ASF::SVN raised an error instead of returning
>> nil when it cannot find the location.
>> It's not always easy to debug the errors that occur when the SVN dir
>> is not found.
>
> So... it would be helpful if there were two versions of the method to
> find an SVN directory, one where it is important that the directory is
> found, and one where the directory is optional (per above). I've now
> made ASF::SVN.find! and ASF::SVN.find (with and without an exclamation
> point) methods, and made the [] operator call the find! method for
> now.
Great.
> - Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Wed, Mar 2, 2016 at 9:22 AM, sebb <se...@gmail.com> wrote:
> On 2 March 2016 at 13:52, Sam Ruby <ru...@intertwingly.net> wrote:
>> On Wed, Mar 2, 2016 at 8:09 AM, sebb <se...@gmail.com> wrote:
>>> Some of the code needs access to a single file in SVN, e.g.
>>>
>>> https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings.xml
>>>
>>> AFAIK there are no other items needed in the same part of SVN.
>>>
>>> Yet one has to checkout the parent directory and keep it updated.
>>>
>>> This is a nuisance for testing purposes.
>>>
>>> For SVN files that don't need authentication, it is easy enough to
>>> fetch the file using HTTP(S) and cache it locally. For subsequent
>>> requests, the cache date can be used in an If-Modified request.
>>> The last checked date can also be saved to avoid checking the file too often.
>>>
>>> This technique has been used in reporter.a.o for checking the chair
>>> index in foundation/index.mdtext:
>>>
>>> https://svn.apache.org/repos/asf/comdev/reporter.apache.org/trunk/scripts/parsepmcs.py
>>>
>>> Might be worth cosidering re-implementing the python caching code used there.
>>
>> As very few of the resources accessed by whimsy are public, that would
>> only solve a small part of the problem.
>
> Which in itself is an issue - should we be encouraging downloading all
> these resources for testing purposes?
>
> In particular, it seems wrong to have to download all the ICLAs.
Ideally, applications would only make optional use of such
repositories - and would check to see if ASF::SVN returned nil or not.
>> Given that the resources needed are declared in
>> https://github.com/apache/whimsy/blob/master/repository.yml, perhaps a
>> more complete solution would be to use the 'svn:update' task defined
>> in https://github.com/apache/whimsy/blob/master/Rakefile#L66 ?
>>
>> At the moment, that task is assuming that you do *NOT* override :svn:
>> in your .whimsy config file. Perhaps it should handle that case?
>
> Better would be to have a function to report which entries are
> missing, so the user can decide which they want to download based on
> what they are testing.
>
> Maybe also with a function to report which are out of date.
I've roughed in such a tool. You will need to do a "bundle update" or
otherwise get the latest Wunderbar (1.0.21) to test it out locally.
Then run:
ruby www/status/svn.cgi
And then visit http://localhost:4567/
Checkout is not yet implemented (it would need to prompt you for a destination).
This tool is deployed on whimsy, where it runs in read-only mode:
https://whimsy.apache.org/status/svn
> It might be useful if ASF::SVN raised an error instead of returning
> nil when it cannot find the location.
> It's not always easy to debug the errors that occur when the SVN dir
> is not found.
So... it would be helpful if there were two versions of the method to
find an SVN directory, one where it is important that the directory is
found, and one where the directory is optional (per above). I've now
made ASF::SVN.find! and ASF::SVN.find (with and without an exclamation
point) methods, and made the [] operator call the find! method for
now.
- Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by sebb <se...@gmail.com>.
On 2 March 2016 at 13:52, Sam Ruby <ru...@intertwingly.net> wrote:
> On Wed, Mar 2, 2016 at 8:09 AM, sebb <se...@gmail.com> wrote:
>> Some of the code needs access to a single file in SVN, e.g.
>>
>> https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings.xml
>>
>> AFAIK there are no other items needed in the same part of SVN.
>>
>> Yet one has to checkout the parent directory and keep it updated.
>>
>> This is a nuisance for testing purposes.
>>
>> For SVN files that don't need authentication, it is easy enough to
>> fetch the file using HTTP(S) and cache it locally. For subsequent
>> requests, the cache date can be used in an If-Modified request.
>> The last checked date can also be saved to avoid checking the file too often.
>>
>> This technique has been used in reporter.a.o for checking the chair
>> index in foundation/index.mdtext:
>>
>> https://svn.apache.org/repos/asf/comdev/reporter.apache.org/trunk/scripts/parsepmcs.py
>>
>> Might be worth cosidering re-implementing the python caching code used there.
>
> As very few of the resources accessed by whimsy are public, that would
> only solve a small part of the problem.
Which in itself is an issue - should we be encouraging downloading all
these resources for testing purposes?
In particular, it seems wrong to have to download all the ICLAs.
> Given that the resources needed are declared in
> https://github.com/apache/whimsy/blob/master/repository.yml, perhaps a
> more complete solution would be to use the 'svn:update' task defined
> in https://github.com/apache/whimsy/blob/master/Rakefile#L66 ?
>
> At the moment, that task is assuming that you do *NOT* override :svn:
> in your .whimsy config file. Perhaps it should handle that case?
Better would be to have a function to report which entries are
missing, so the user can decide which they want to download based on
what they are testing.
Maybe also with a function to report which are out of date.
It might be useful if ASF::SVN raised an error instead of returning
nil when it cannot find the location.
It's not always easy to debug the errors that occur when the SVN dir
is not found.
> - Sam Ruby
Re: Access to single SVN items - implement caching?
Posted by Sam Ruby <ru...@intertwingly.net>.
On Wed, Mar 2, 2016 at 8:09 AM, sebb <se...@gmail.com> wrote:
> Some of the code needs access to a single file in SVN, e.g.
>
> https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings.xml
>
> AFAIK there are no other items needed in the same part of SVN.
>
> Yet one has to checkout the parent directory and keep it updated.
>
> This is a nuisance for testing purposes.
>
> For SVN files that don't need authentication, it is easy enough to
> fetch the file using HTTP(S) and cache it locally. For subsequent
> requests, the cache date can be used in an If-Modified request.
> The last checked date can also be saved to avoid checking the file too often.
>
> This technique has been used in reporter.a.o for checking the chair
> index in foundation/index.mdtext:
>
> https://svn.apache.org/repos/asf/comdev/reporter.apache.org/trunk/scripts/parsepmcs.py
>
> Might be worth cosidering re-implementing the python caching code used there.
As very few of the resources accessed by whimsy are public, that would
only solve a small part of the problem.
Given that the resources needed are declared in
https://github.com/apache/whimsy/blob/master/repository.yml, perhaps a
more complete solution would be to use the 'svn:update' task defined
in https://github.com/apache/whimsy/blob/master/Rakefile#L66 ?
At the moment, that task is assuming that you do *NOT* override :svn:
in your .whimsy config file. Perhaps it should handle that case?
- Sam Ruby