Re: Prep'ing whimsy-vm3

[Sam Ruby <ru...@apache.org>]

TL;DR: whimsy3.apache.org is now operational.  Details below.

On 2016-04-23 10:49, Sam Ruby <ru...@intertwingly.net> wrote: 
> We just got settled into whimsy-vm2, and I've already started working on 
> whimsy-vm3.
> 
> The precipitating factor was 
> https://issues.apache.org/jira/browse/INFRA-11723; where the solution 
> was to cache content that was statically being served by Apache httpd to 
> another machine on another network... where it would be statically 
> served by Apache httpd.

I wonder if it would be worth undoing this?  Either that or changing the code to try to fetch from whimsy and then fail back to a local cached copy on failure?

> This supports the notion that the problem is either the network or the 
> proxy.  So I requested and obtained a new VM on a different colo 
> facility and network:
> 
> https://issues.apache.org/jira/browse/INFRA-11725
> 
> As a testament to puppet, the site is already sorta there 
> (http://whimsy3.apache.org), the largest thing missing is the svn 
> checkouts that the code requires.

Puppet really made this easy.  Only differences I noted:

1) on whimsy-vm3; most of the space is on /x1; so I changed the directory layout

2) no proxy means that a signed cert is needed, and the apache configuration tweaked accordingly

3) the HOME directory is different for CGIs; this affects the secretarial workbench.  Might be a consequence of #2 above.

> Note that for the moment, the site is accessed via http instead of 
> https; that's because this vm is NOT currently accessed via a proxy. 
> I'd like to keep it that way at least initially so that can more easily 
> determine where the actual problem lies.  I'm working out the mechanics 
> of obtaining an SSL cert for this, and that should be resolved in a few 
> days.

This is indeed fixed now.

> If all goes well, I'll ask that the DNS entry for whimsy.apache.org to 
> be updated to point to whimsy-vm3 before the next board meeting, and 
> then to release whimsy-vm2 after that point.

As this point, I'd rather wait until after the next meeting to make this switch; I'll be using whimsy3 during the board meeting, and will be encouraging others to do likewise.

If anybody has any remaining need for whimsy-vm2 please speak up now; otherwise it will likely go away by the end of the month.

> The original whimsy-vm still hosts some unrelated tools (e.g., 
> infra.apache.org and etherpad), and that would need to be addressed 
> before that machine is released.

I have no further need for this vm; but haven't spent any cycles trying to figure out who else might.

- Sam Ruby

Re: Prep'ing whimsy-vm3

[sebb <se...@gmail.com>]

On 14 May 2016 at 14:19, Sam Ruby <ru...@intertwingly.net> wrote:
> On Sat, May 14, 2016 at 4:55 AM, sebb <se...@gmail.com> wrote:
>> On 13 May 2016 at 17:25, Sam Ruby <ru...@apache.org> wrote:
>>> TL;DR: whimsy3.apache.org is now operational.  Details below.
>>>
>>> On 2016-04-23 10:49, Sam Ruby <ru...@intertwingly.net> wrote:
>>>> We just got settled into whimsy-vm2, and I've already started working on
>>>> whimsy-vm3.
>>>>
>>>> The precipitating factor was
>>>> https://issues.apache.org/jira/browse/INFRA-11723; where the solution
>>>> was to cache content that was statically being served by Apache httpd to
>>>> another machine on another network... where it would be statically
>>>> served by Apache httpd.
>>>
>>> I wonder if it would be worth undoing this?  Either that or changing the code to try to fetch from whimsy and then fail back to a local cached copy on failure?
>>
>> Although the problem with *really* slow responses was a temporary
>> issue, it's definitely slower using remote access.
>>
>> Compare the current local access:
>> http://home.apache.org/phonebook.html
>>
>> with whimsy access:
>> http://home.apache.org/whimsybook.html
>> or
>> http://home.apache.org/whimsy3book.html
>>
>> Clear the browser cache to see the full effect.
>
> whimsy3book is equally as speedy for me.

I agree whimsy3 is a lot better than vm2, however it was still
noticeably slower for me when I tested initially.
Not sure why.

Further tests show that the whimsy3 times are very similar to the
'local' version.

> The concept of "remote" makes no sense to me in this context.  The
> phonebook consists of a static HTML page, a static JS page, and a
> small number of static JSON pages.  All are served using Apache httpd
> running on ASF infrastructure.  One host has more recent data, another
> has slightly older data.  The speed should be no different.

I see what you mean now.

However there is a slight difference in the settings:
home.a.o can use http: whereas whimsy3 uses https:
I assume this will not generally cause a noticeable slowdown.

> If there is a reason why apache httpd serving static files runs faster
> on home.apache.org than it does on whimsy-vm3.apache.org, I would want
> to understand why that is and how it could be fixed.
>
>> Using a fall-back mechanism would inevitably be slower for the cases
>> when Whimsy was not responding promptly.
>> Also I don't know how easy that is to do in Javascript.
>
> There is code in the javascript to fetch pages (getAsyncJSONArray).
> It sends an alert on failure.  It could, instead, try a different URL.
>
> But if people are happy with the phonebook as is, there really is no
> need to change it.

As I see it, the disadvantages with using copies on home.a.o are:
- extra cronjobs are needed
- the data may be up to an hour out of date.
- extra storage needed for the data (trivial)

The advantages are:
- slightly simpler JS code (no need to retry)
- if there is a problem with whimsy3, it does not affect load times.
- the JSON files are present locally for use by other cron jobs.
  They might otherwise need to implement conditional downloads/allow
for temporary unavailability; this is done by wget now.

On average I suppose the data will be 30 mins out of date, plus about
8 minutes for the Whimsy delay between runs.
The max delay will be about 75 minutes.
Is that a problem?
I'm not sure it is serious enough to warrant reverting, but I'm
willing to be persuaded.

>>>> This supports the notion that the problem is either the network or the
>>>> proxy.  So I requested and obtained a new VM on a different colo
>>>> facility and network:
>>>>
>>>> https://issues.apache.org/jira/browse/INFRA-11725
>>>>
>>>> As a testament to puppet, the site is already sorta there
>>>> (http://whimsy3.apache.org), the largest thing missing is the svn
>>>> checkouts that the code requires.
>>>
>>> Puppet really made this easy.  Only differences I noted:
>>>
>>> 1) on whimsy-vm3; most of the space is on /x1; so I changed the directory layout
>>>
>>> 2) no proxy means that a signed cert is needed, and the apache configuration tweaked accordingly
>>>
>>> 3) the HOME directory is different for CGIs; this affects the secretarial workbench.  Might be a consequence of #2 above.
>>>
>>>> Note that for the moment, the site is accessed via http instead of
>>>> https; that's because this vm is NOT currently accessed via a proxy.
>>>> I'd like to keep it that way at least initially so that can more easily
>>>> determine where the actual problem lies.  I'm working out the mechanics
>>>> of obtaining an SSL cert for this, and that should be resolved in a few
>>>> days.
>>>
>>> This is indeed fixed now.
>>>
>>>> If all goes well, I'll ask that the DNS entry for whimsy.apache.org to
>>>> be updated to point to whimsy-vm3 before the next board meeting, and
>>>> then to release whimsy-vm2 after that point.
>>>
>>> As this point, I'd rather wait until after the next meeting to make this switch; I'll be using whimsy3 during the board meeting, and will be encouraging others to do likewise.
>>>
>>> If anybody has any remaining need for whimsy-vm2 please speak up now; otherwise it will likely go away by the end of the month.
>>>
>>>> The original whimsy-vm still hosts some unrelated tools (e.g.,
>>>> infra.apache.org and etherpad), and that would need to be addressed
>>>> before that machine is released.
>>>
>>> I have no further need for this vm; but haven't spent any cycles trying to figure out who else might.
>>
>> OK by me.
>>
>> I think we need to switch off most of the cronjobs on VM2 as soon as
>> we can - the notification messages are currently duplicated.
>> However this probably cannot happen until PingMyBox is talking to VM3.
>
> Instead of turning off the cronjobs, perhaps notifications could
> skipped if `hostname`.include? 'vm2' ?

OK, I'll look at that.

> - Sam Ruby

Re: Prep'ing whimsy-vm3

[Sam Ruby <ru...@intertwingly.net>]

On Sat, May 14, 2016 at 4:55 AM, sebb <se...@gmail.com> wrote:
> On 13 May 2016 at 17:25, Sam Ruby <ru...@apache.org> wrote:
>> TL;DR: whimsy3.apache.org is now operational.  Details below.
>>
>> On 2016-04-23 10:49, Sam Ruby <ru...@intertwingly.net> wrote:
>>> We just got settled into whimsy-vm2, and I've already started working on
>>> whimsy-vm3.
>>>
>>> The precipitating factor was
>>> https://issues.apache.org/jira/browse/INFRA-11723; where the solution
>>> was to cache content that was statically being served by Apache httpd to
>>> another machine on another network... where it would be statically
>>> served by Apache httpd.
>>
>> I wonder if it would be worth undoing this?  Either that or changing the code to try to fetch from whimsy and then fail back to a local cached copy on failure?
>
> Although the problem with *really* slow responses was a temporary
> issue, it's definitely slower using remote access.
>
> Compare the current local access:
> http://home.apache.org/phonebook.html
>
> with whimsy access:
> http://home.apache.org/whimsybook.html
> or
> http://home.apache.org/whimsy3book.html
>
> Clear the browser cache to see the full effect.

whimsy3book is equally as speedy for me.

The concept of "remote" makes no sense to me in this context.  The
phonebook consists of a static HTML page, a static JS page, and a
small number of static JSON pages.  All are served using Apache httpd
running on ASF infrastructure.  One host has more recent data, another
has slightly older data.  The speed should be no different.

If there is a reason why apache httpd serving static files runs faster
on home.apache.org than it does on whimsy-vm3.apache.org, I would want
to understand why that is and how it could be fixed.

> Using a fall-back mechanism would inevitably be slower for the cases
> when Whimsy was not responding promptly.
> Also I don't know how easy that is to do in Javascript.

There is code in the javascript to fetch pages (getAsyncJSONArray).
It sends an alert on failure.  It could, instead, try a different URL.

But if people are happy with the phonebook as is, there really is no
need to change it.

>>> This supports the notion that the problem is either the network or the
>>> proxy.  So I requested and obtained a new VM on a different colo
>>> facility and network:
>>>
>>> https://issues.apache.org/jira/browse/INFRA-11725
>>>
>>> As a testament to puppet, the site is already sorta there
>>> (http://whimsy3.apache.org), the largest thing missing is the svn
>>> checkouts that the code requires.
>>
>> Puppet really made this easy.  Only differences I noted:
>>
>> 1) on whimsy-vm3; most of the space is on /x1; so I changed the directory layout
>>
>> 2) no proxy means that a signed cert is needed, and the apache configuration tweaked accordingly
>>
>> 3) the HOME directory is different for CGIs; this affects the secretarial workbench.  Might be a consequence of #2 above.
>>
>>> Note that for the moment, the site is accessed via http instead of
>>> https; that's because this vm is NOT currently accessed via a proxy.
>>> I'd like to keep it that way at least initially so that can more easily
>>> determine where the actual problem lies.  I'm working out the mechanics
>>> of obtaining an SSL cert for this, and that should be resolved in a few
>>> days.
>>
>> This is indeed fixed now.
>>
>>> If all goes well, I'll ask that the DNS entry for whimsy.apache.org to
>>> be updated to point to whimsy-vm3 before the next board meeting, and
>>> then to release whimsy-vm2 after that point.
>>
>> As this point, I'd rather wait until after the next meeting to make this switch; I'll be using whimsy3 during the board meeting, and will be encouraging others to do likewise.
>>
>> If anybody has any remaining need for whimsy-vm2 please speak up now; otherwise it will likely go away by the end of the month.
>>
>>> The original whimsy-vm still hosts some unrelated tools (e.g.,
>>> infra.apache.org and etherpad), and that would need to be addressed
>>> before that machine is released.
>>
>> I have no further need for this vm; but haven't spent any cycles trying to figure out who else might.
>
> OK by me.
>
> I think we need to switch off most of the cronjobs on VM2 as soon as
> we can - the notification messages are currently duplicated.
> However this probably cannot happen until PingMyBox is talking to VM3.

Instead of turning off the cronjobs, perhaps notifications could
skipped if `hostname`.include? 'vm2' ?

- Sam Ruby

Re: Prep'ing whimsy-vm3

[sebb <se...@gmail.com>]

On 13 May 2016 at 17:25, Sam Ruby <ru...@apache.org> wrote:
> TL;DR: whimsy3.apache.org is now operational.  Details below.
>
> On 2016-04-23 10:49, Sam Ruby <ru...@intertwingly.net> wrote:
>> We just got settled into whimsy-vm2, and I've already started working on
>> whimsy-vm3.
>>
>> The precipitating factor was
>> https://issues.apache.org/jira/browse/INFRA-11723; where the solution
>> was to cache content that was statically being served by Apache httpd to
>> another machine on another network... where it would be statically
>> served by Apache httpd.
>
> I wonder if it would be worth undoing this?  Either that or changing the code to try to fetch from whimsy and then fail back to a local cached copy on failure?

Although the problem with *really* slow responses was a temporary
issue, it's definitely slower using remote access.

Compare the current local access:
http://home.apache.org/phonebook.html

with whimsy access:
http://home.apache.org/whimsybook.html
or
http://home.apache.org/whimsy3book.html


Clear the browser cache to see the full effect.

Using a fall-back mechanism would inevitably be slower for the cases
when Whimsy was not responding promptly.
Also I don't know how easy that is to do in Javascript.

>> This supports the notion that the problem is either the network or the
>> proxy.  So I requested and obtained a new VM on a different colo
>> facility and network:
>>
>> https://issues.apache.org/jira/browse/INFRA-11725
>>
>> As a testament to puppet, the site is already sorta there
>> (http://whimsy3.apache.org), the largest thing missing is the svn
>> checkouts that the code requires.
>
> Puppet really made this easy.  Only differences I noted:
>
> 1) on whimsy-vm3; most of the space is on /x1; so I changed the directory layout
>
> 2) no proxy means that a signed cert is needed, and the apache configuration tweaked accordingly
>
> 3) the HOME directory is different for CGIs; this affects the secretarial workbench.  Might be a consequence of #2 above.
>
>> Note that for the moment, the site is accessed via http instead of
>> https; that's because this vm is NOT currently accessed via a proxy.
>> I'd like to keep it that way at least initially so that can more easily
>> determine where the actual problem lies.  I'm working out the mechanics
>> of obtaining an SSL cert for this, and that should be resolved in a few
>> days.
>
> This is indeed fixed now.
>
>> If all goes well, I'll ask that the DNS entry for whimsy.apache.org to
>> be updated to point to whimsy-vm3 before the next board meeting, and
>> then to release whimsy-vm2 after that point.
>
> As this point, I'd rather wait until after the next meeting to make this switch; I'll be using whimsy3 during the board meeting, and will be encouraging others to do likewise.
>
> If anybody has any remaining need for whimsy-vm2 please speak up now; otherwise it will likely go away by the end of the month.
>
>> The original whimsy-vm still hosts some unrelated tools (e.g.,
>> infra.apache.org and etherpad), and that would need to be addressed
>> before that machine is released.
>
> I have no further need for this vm; but haven't spent any cycles trying to figure out who else might.

OK by me.

I think we need to switch off most of the cronjobs on VM2 as soon as
we can - the notification messages are currently duplicated.
However this probably cannot happen until PingMyBox is talking to VM3.

> - Sam Ruby