You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Dave Fisher <da...@comcast.net> on 2012/08/15 16:00:06 UTC
Registration and Update Services - What Will Be The Load?
Hi,
Apologies to the Apache Infra team, the load caused by implementing INFRA-5112 caused trouble.
The changes to update*.services.openoffice.org to point to www.openoffice.org was reverted.
This was due to added load on the Apache Infrastructure.
Before we can proceed with INFRA-5112 and INFRA-5144 we absolutely must have some estimates about the volume of requests that will be received.
Regards,
Dave
Re: Registration and Update Services - What Will Be The Load?
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 15, 2012 at 10:00 AM, Dave Fisher <da...@comcast.net> wrote:
> Hi,
>
> Apologies to the Apache Infra team, the load caused by implementing INFRA-5112 caused trouble.
>
> The changes to update*.services.openoffice.org to point to www.openoffice.org was reverted.
>
> This was due to added load on the Apache Infrastructure.
>
Can you be specific? Are they seeing unusual load from the change?
or are they just not willing to make the change unless we first
provide them an estimate?
If you recall, we made this change for OOo 3.3.0, OOo 3.2.1 and OOo
3.2.0 already, and we never provided any estimates. I don't think we
received any complaints about those. Have you heard any load
concerns?
In any case, the general trend we see for the upgrades that are
enabled is around 40,000/day. But there was a peak of around
60,000/day for a couple of weeks when they were first enabled. That
is the number who have actually updated. We don't know what % of
users dismiss the update notification but don't disable it, and so
receive such a notification every week. I assume this is small.
Also, generally, we have diminishing numbers as we go back to older
versions. For example, we've seen around 8K/day for US/Windows/3.3.0
upgrades, but only around 1K/day for OOo 3.2.1 upgrades, and around
500/day for OOo 3.2.0 upgrades. So this would suggest that enabling
the older versions will pick up a few, but I wouldn't expect huge
numbers.
> Before we can proceed with INFRA-5112 and INFRA-5144 we absolutely must have some estimates about the volume of requests that will be received.
>
Hopefully the above helps.
-Rob
> Regards,
> Dave
>
Re: Registration and Update Services - What Will Be The Load?
Posted by Kay Schenk <ka...@gmail.com>.
On Wed, Aug 15, 2012 at 9:26 AM, Dave Fisher <da...@comcast.net> wrote:
>
> On Aug 15, 2012, at 8:50 AM, Rob Weir wrote:
>
> > On Wed, Aug 15, 2012 at 10:32 AM, Oliver-Rainer Wittmann
> > <or...@googlemail.com> wrote:
> >> Hi,
> >>
> >>
> >> On 15.08.2012 16:00, Dave Fisher wrote:
> >>>
> >>> Hi,
> >>>
> >>> Apologies to the Apache Infra team, the load caused by implementing
> >>> INFRA-5112 caused trouble.
> >>>
> >>> The changes to update*.services.openoffice.org to point to
> >>> www.openoffice.org
> >>> was reverted.
> >>>
> >>> This was due to added load on the Apache Infrastructure.
> >>>
> >>> Before we can proceed with INFRA-5112 and INFRA-5144 we absolutely must
> >>> have
> >>> some estimates about the volume of requests that will be received.
> >>>
> >>
> >> When I saw the problems on www.openoffice.org in the morning I already
> >> suggested that this might be related to the established redirects. I
> >> reported my assumption at #asfinfra in the morning.
> >> If I remembering it correct, Kay S. and Joe S. already observed
> something
> >> like this earlier this year.
> >>
>
Well as we kept going backward from 3.3 on down, I was wondering if we
might run into problems again...
The problem with the 3.0 update was that the "call" was a POST, which is
disallowed on Apache, and yes, brought the Apache web server (all of it,
not just the OO site) to its knees, instead of a GET for the update XML.
So, based on this notice, it seems 3.1 might use a POST also? If so,
there's nothing we can do vis a vis enacting this update for end users.
>> Unfortunately, I can not provide any volume.
> >> In the past more than 100 million download of OpenOffice.org package had
> >> been counted by the OpenOffice.org community. I do not know how much OOo
> >> installation are really active and how the distribution between the
> >> different versions are.
> >>
> >> What I know is the following:
> >> (1) update38.services.openoffice.org is used by OOo 3.4 Beta and
> released
> >> AOO 3.4 - a redirect for it has been established at 2012-05-21 and the
> >> traffic can be handled.
> >> (2) update36.services.openoffice.org is used by OOo 3.3 - a redirect
> for it
> >> has been established at 2012-06-04 and the traffic can be handled.
> >> (3) update35|34.services.openoffice.org is used by OOo 3.2.1 and OOo
> 3.2 -
> >> redirects for then have been established at 2012-07-12 and the traffic
> can
> >> be handled.
> >> (4) update33.services.openoffice.org is _not_ used - a redirect is
> _not_
> >> necessary. Does occur any traffic on this URL?
> >> (5) update32.services.openoffice.org is used by OOo 3.1.1 and OOo 3.1
> - a
> >> redirect has been requested and was established today. Due to the server
> >> load it has been reverted. Is the traffic data available?
> >> (6) update31.services.openoffice.org is _not_ used - a redirect is
> _not_
> >> necessary. Does occur any traffic on this URL?
> >> (7) update30.services.openoffice.org is used by OOo 3.0.1 and OOo 3.0.
> Does
> >> occur any traffic on this URL?
> >> (8) update.services.openoffice.org seems to be used by OOo 2.x version
> (at
> >> least my test installation of OOo 2.2 uses it). Is the traffic data
> >> available? When I remember it correct Kay S. and Joe S. observed the
> above
> >> mentioned problems earlier this year with this URL.
> >>
> >> Is it possible that somebody from the Apache Infrastructure can provide
> a
> >> view on which URL the traffic load was soo high that the servers got in
> >> trouble?
> >>
> >
> > That would be good to know. Once we know we could verify behavior
> > with a change to a local hosts file. One scenario that could
> > conceivably cause a problem would be if some old version of OOo
> > behaved badly when it gets a 404 error, such as getting into a retry
> > loop. We know this did not happen for OOo 3.3.0, 3.2.1 or 3.2.0.
> > But it is worth confirming with earlier versions, if the HTTP logs
> > suggest this is happening.
>
> We'll have access to full logs tonight / tomorrow.
>
> Regards,
> Dave
>
> >
> > -Rob
> >
> >> Best regards, Oliver.
>
>
--
----------------------------------------------------------------------------------------
MzK
"Never express yourself more clearly than you are able to think."
--
Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Dave Fisher <da...@comcast.net>.
On Aug 15, 2012, at 8:50 AM, Rob Weir wrote:
> On Wed, Aug 15, 2012 at 10:32 AM, Oliver-Rainer Wittmann
> <or...@googlemail.com> wrote:
>> Hi,
>>
>>
>> On 15.08.2012 16:00, Dave Fisher wrote:
>>>
>>> Hi,
>>>
>>> Apologies to the Apache Infra team, the load caused by implementing
>>> INFRA-5112 caused trouble.
>>>
>>> The changes to update*.services.openoffice.org to point to
>>> www.openoffice.org
>>> was reverted.
>>>
>>> This was due to added load on the Apache Infrastructure.
>>>
>>> Before we can proceed with INFRA-5112 and INFRA-5144 we absolutely must
>>> have
>>> some estimates about the volume of requests that will be received.
>>>
>>
>> When I saw the problems on www.openoffice.org in the morning I already
>> suggested that this might be related to the established redirects. I
>> reported my assumption at #asfinfra in the morning.
>> If I remembering it correct, Kay S. and Joe S. already observed something
>> like this earlier this year.
>>
>> Unfortunately, I can not provide any volume.
>> In the past more than 100 million download of OpenOffice.org package had
>> been counted by the OpenOffice.org community. I do not know how much OOo
>> installation are really active and how the distribution between the
>> different versions are.
>>
>> What I know is the following:
>> (1) update38.services.openoffice.org is used by OOo 3.4 Beta and released
>> AOO 3.4 - a redirect for it has been established at 2012-05-21 and the
>> traffic can be handled.
>> (2) update36.services.openoffice.org is used by OOo 3.3 - a redirect for it
>> has been established at 2012-06-04 and the traffic can be handled.
>> (3) update35|34.services.openoffice.org is used by OOo 3.2.1 and OOo 3.2 -
>> redirects for then have been established at 2012-07-12 and the traffic can
>> be handled.
>> (4) update33.services.openoffice.org is _not_ used - a redirect is _not_
>> necessary. Does occur any traffic on this URL?
>> (5) update32.services.openoffice.org is used by OOo 3.1.1 and OOo 3.1 - a
>> redirect has been requested and was established today. Due to the server
>> load it has been reverted. Is the traffic data available?
>> (6) update31.services.openoffice.org is _not_ used - a redirect is _not_
>> necessary. Does occur any traffic on this URL?
>> (7) update30.services.openoffice.org is used by OOo 3.0.1 and OOo 3.0. Does
>> occur any traffic on this URL?
>> (8) update.services.openoffice.org seems to be used by OOo 2.x version (at
>> least my test installation of OOo 2.2 uses it). Is the traffic data
>> available? When I remember it correct Kay S. and Joe S. observed the above
>> mentioned problems earlier this year with this URL.
>>
>> Is it possible that somebody from the Apache Infrastructure can provide a
>> view on which URL the traffic load was soo high that the servers got in
>> trouble?
>>
>
> That would be good to know. Once we know we could verify behavior
> with a change to a local hosts file. One scenario that could
> conceivably cause a problem would be if some old version of OOo
> behaved badly when it gets a 404 error, such as getting into a retry
> loop. We know this did not happen for OOo 3.3.0, 3.2.1 or 3.2.0.
> But it is worth confirming with earlier versions, if the HTTP logs
> suggest this is happening.
We'll have access to full logs tonight / tomorrow.
Regards,
Dave
>
> -Rob
>
>> Best regards, Oliver.
Re: Registration and Update Services - What Will Be The Load?
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 15, 2012 at 10:32 AM, Oliver-Rainer Wittmann
<or...@googlemail.com> wrote:
> Hi,
>
>
> On 15.08.2012 16:00, Dave Fisher wrote:
>>
>> Hi,
>>
>> Apologies to the Apache Infra team, the load caused by implementing
>> INFRA-5112 caused trouble.
>>
>> The changes to update*.services.openoffice.org to point to
>> www.openoffice.org
>> was reverted.
>>
>> This was due to added load on the Apache Infrastructure.
>>
>> Before we can proceed with INFRA-5112 and INFRA-5144 we absolutely must
>> have
>> some estimates about the volume of requests that will be received.
>>
>
> When I saw the problems on www.openoffice.org in the morning I already
> suggested that this might be related to the established redirects. I
> reported my assumption at #asfinfra in the morning.
> If I remembering it correct, Kay S. and Joe S. already observed something
> like this earlier this year.
>
> Unfortunately, I can not provide any volume.
> In the past more than 100 million download of OpenOffice.org package had
> been counted by the OpenOffice.org community. I do not know how much OOo
> installation are really active and how the distribution between the
> different versions are.
>
> What I know is the following:
> (1) update38.services.openoffice.org is used by OOo 3.4 Beta and released
> AOO 3.4 - a redirect for it has been established at 2012-05-21 and the
> traffic can be handled.
> (2) update36.services.openoffice.org is used by OOo 3.3 - a redirect for it
> has been established at 2012-06-04 and the traffic can be handled.
> (3) update35|34.services.openoffice.org is used by OOo 3.2.1 and OOo 3.2 -
> redirects for then have been established at 2012-07-12 and the traffic can
> be handled.
> (4) update33.services.openoffice.org is _not_ used - a redirect is _not_
> necessary. Does occur any traffic on this URL?
> (5) update32.services.openoffice.org is used by OOo 3.1.1 and OOo 3.1 - a
> redirect has been requested and was established today. Due to the server
> load it has been reverted. Is the traffic data available?
> (6) update31.services.openoffice.org is _not_ used - a redirect is _not_
> necessary. Does occur any traffic on this URL?
> (7) update30.services.openoffice.org is used by OOo 3.0.1 and OOo 3.0. Does
> occur any traffic on this URL?
> (8) update.services.openoffice.org seems to be used by OOo 2.x version (at
> least my test installation of OOo 2.2 uses it). Is the traffic data
> available? When I remember it correct Kay S. and Joe S. observed the above
> mentioned problems earlier this year with this URL.
>
> Is it possible that somebody from the Apache Infrastructure can provide a
> view on which URL the traffic load was soo high that the servers got in
> trouble?
>
That would be good to know. Once we know we could verify behavior
with a change to a local hosts file. One scenario that could
conceivably cause a problem would be if some old version of OOo
behaved badly when it gets a 404 error, such as getting into a retry
loop. We know this did not happen for OOo 3.3.0, 3.2.1 or 3.2.0.
But it is worth confirming with earlier versions, if the HTTP logs
suggest this is happening.
-Rob
> Best regards, Oliver.
Re: Registration and Update Services - What Will Be The Load?
Posted by Kay Schenk <ka...@gmail.com>.
On 08/17/2012 03:55 AM, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 16.08.2012 14:42, Oliver-Rainer Wittmann wrote:
>> Hi
>>
>> On 15.08.2012 19:05, Daniel Shahaf wrote:
>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>>>> Is it possible that somebody from the Apache Infrastructure can
>>>> provide a view on which URL the traffic load was soo high that the
>>>> servers got in trouble?
>>>>
>>>
>>> POST requests to /ProductUpdateService/check.Update file
>>>
>>
>> I am currently investigating the server logs and will post my results
>> later.
>>
>
> The trouble causing HTTP POST requests found in the server logs does not
> seem to be related to the update function in former OOo versions.
> I have investigated the update function HTTP requests from various
> former OOo versions in the server logs. All of them provide content for
> the HTTP UserAgent field. All of found HTTP POST request does not have
> such data.
> Looking at some sample HTTP requests from one IP address reveals that
> the HTTP requests from the update function are unrelated to the HTTP
> POST requests.
> I investigated the HTTP traffic of update function from former OOo
> versions (OOo 2.2, OOo 2.3, OOo 2.3.1, OOo 2.4, OOo 3.x). None of these
> showed an HTTP POST request.
> I evaluated some server logs from days before the established redirects
> from INFRA-5112. Here, I could not found any of such HTTP POST requests.
> My conclusion is that there were another function in OOo which causes
> these HTTP POST requests. OOo 3.2, OOo 3.2.1, OOo 3.3 does not seem to
> have such a function, because the redirects for their update functions
> had been established a couple of weeks ago and I did not found any HTTP
> POST requests.
how curious... thanks for the update and continued good luck with the
investigation
>
> In order to figure out what is going on here, I will follow up with
> Apache infrastructure. I will report on ooo-dev, when I know more.
>
> Best regards, Oliver.
--
------------------------------------------------------------------------
MzK
"Never express yourself more clearly than you are able to think."
-- Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 16.08.2012 14:42, Oliver-Rainer Wittmann wrote:
> Hi
>
> On 15.08.2012 19:05, Daniel Shahaf wrote:
>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>>> Is it possible that somebody from the Apache Infrastructure can
>>> provide a view on which URL the traffic load was soo high that the
>>> servers got in trouble?
>>>
>>
>> POST requests to /ProductUpdateService/check.Update file
>>
>
> I am currently investigating the server logs and will post my results later.
>
The trouble causing HTTP POST requests found in the server logs does not seem to
be related to the update function in former OOo versions.
I have investigated the update function HTTP requests from various former OOo
versions in the server logs. All of them provide content for the HTTP UserAgent
field. All of found HTTP POST request does not have such data.
Looking at some sample HTTP requests from one IP address reveals that the HTTP
requests from the update function are unrelated to the HTTP POST requests.
I investigated the HTTP traffic of update function from former OOo versions (OOo
2.2, OOo 2.3, OOo 2.3.1, OOo 2.4, OOo 3.x). None of these showed an HTTP POST
request.
I evaluated some server logs from days before the established redirects from
INFRA-5112. Here, I could not found any of such HTTP POST requests.
My conclusion is that there were another function in OOo which causes these HTTP
POST requests. OOo 3.2, OOo 3.2.1, OOo 3.3 does not seem to have such a
function, because the redirects for their update functions had been established
a couple of weeks ago and I did not found any HTTP POST requests.
In order to figure out what is going on here, I will follow up with Apache
infrastructure. I will report on ooo-dev, when I know more.
Best regards, Oliver.
Re: Registration and Update Services - What Will Be The Load?
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi
On 15.08.2012 19:05, Daniel Shahaf wrote:
> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>> Is it possible that somebody from the Apache Infrastructure can
>> provide a view on which URL the traffic load was soo high that the
>> servers got in trouble?
>>
>
> POST requests to /ProductUpdateService/check.Update file
>
I am currently investigating the server logs and will post my results later.
Best regards, Oliver.
Re: Registration and Update Services - What Will Be The Load?
Posted by Dave Fisher <da...@comcast.net>.
On Aug 15, 2012, at 6:53 PM, Rob Weir wrote:
> On Wed, Aug 15, 2012 at 7:31 PM, Kay Schenk <ka...@gmail.com> wrote:
>> On Wed, Aug 15, 2012 at 4:04 PM, Rob Weir <ro...@apache.org> wrote:
>>
>>> On Wed, Aug 15, 2012 at 5:29 PM, Kay Schenk <ka...@gmail.com> wrote:
>>>> On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <ro...@apache.org> wrote:
>>>>
>>>>> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net>
>>>>> wrote:
>>>>>>
>>>>>> On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
>>>>>>
>>>>>>> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
>>>>>>>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <
>>> d.s@daniel.shahaf.name>
>>>>> wrote:
>>>>>>>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21
>>> +0200:
>>>>>>>>>> Is it possible that somebody from the Apache Infrastructure can
>>>>>>>>>> provide a view on which URL the traffic load was soo high that the
>>>>>>>>>> servers got in trouble?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> POST requests to /ProductUpdateService/check.Update file
>>>>>>>>>
>>>>>>>>
>>>>>>>> For which subdomain, which UpdateXX.openoffice.org ?
>>>>>>>
>>>>>>> The access log doesn't say, and the error log has
>>>>>>>
>>>>>>> % fgrep /ProductUpdateService/check.Update error_log | sed -e
>>>>> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
>>>>>>>
>>>>>>> EU:
>>>>>>> 232046 update30
>>>>>>> 35548 update34
>>>>>>> 76543 update35
>>>>>>>
>>>>>>>
>>>>>>> US:
>>>>>>> 198996 update30
>>>>>>> 33450 update34
>>>>>>> 71117 update35
>>>>>>> 0 update36
>>>>>>
>>>>>> We don't see update32 because those do not get redirected in the same
>>>>> way because there is no ooo-site/trunk/content/projects/update32
>>>>>
>>>>
>>>> uh oh...this should have been setup before and Oliver said he requested
>>>> this in the first post here.
>>>>
>>>> And you're now saying that all the previous ones have been reverted?
>>>>
>>>> I think we were OK until this last one, right? update32?
>>>>
>>>> I think the others should be re-established as they weren't causing
>>>> problems, were they?
>>>>
>>>> The thing is unless we go back to the code for OOo 3.1, we don't know
>>> what
>>>> it's doing.
>>>>
>>>> When I asked about this when we had issues for OOo 3.0, I was told it was
>>>> fixed in OOo 3.2, so maybe 3.1. has the same issues?
>>>>
>>>>
>>>>
>>>>>
>>>>>> ./update/aoo341/check.Update
>>>>>> ./update/ProductUpdateService/check.Update
>>>>>> ./update30/ProductUpdateService/check.Update
>>>>>> ./update34/ProductUpdateService/check.Update
>>>>>> ./update34/ProductUpdateService/test.Update
>>>>>> ./update35/ProductUpdateService/check.Update
>>>>>> ./update35/ProductUpdateService/test.Update
>>>>>> ./update36/ProductUpdateService/check.Update
>>>>>> ./update38/ProductUpdateService/check.Update
>>>>>>
>>>>>> It looks like 34 and 35 have been trouble, but not as bad as 30.
>>>>>>
>>>>>
>>>>> But haven't 34 and 35 been in production since early July? We've
>>>>> certainly seen plenty of downloads triggered by them. They should not
>>>>> be giving any errors, since the requests resolve to files on our site.
>>>>>
>>>>> I wonder, could the errors in those be caused by the outage caused by
>>>>> the errors in update30?
>>>>>
>>>>
>>>> Rob...update 30 is completely out of the question, and we simply can not
>>>> do it, and reverted it within hours when I first requested it.
>>>>
>>>
>>> What is the issue with update 30? The fact that it does a POST? I
>>> don't that would rule it out altogether. But we would need to treat
>>> it specially. For example, we could redirect to an isolated server,
>>> at Apache or outside, that is able/willing to handle it. If we run it
>>> for a month or two we should get the bulk of the upgrades.
>>>
>>> Or was there some other issue?
>>>
>>
>> The Apache web server, of which AOO is a part, does not allow POSTs so when
>> I had infra enable this and redirect the old update30 to the web server, it
>> caused MANY errors in a very short period of time (about an hour) and Joe
>> reverted it rather quickly. THis was like back in early March or
>> something when I was playing around. The update feed itself didn't even DO
>> anything but redirect them to a URL (in theory) it was the POST in the code
>> for OOo 3.0 that caused all the havoc. When I inquired about this on this
>> list, I was told yes, this WAS the case for 3.0 but had been fixed with, I
>> thought 3.2.
>>
>> Anyway, as far as I know, this is the only issue.
>>
>> I was pretty wary initially about running the feed through the web server
>> but was told we should be fine (this after Joe reverted the update30 in
>> March) -- and I think we have been for the most part. But, yes, we need
>> another box with a web server that would accept POSTs to deal with this --
>> 3.0, and it looks like 3.1.
>>
>
> OK. That's the impression I had as well -- POST was disabled in
> general on ASF servers. We're not going to be able to change that.
> But the traffic from permitted POSTs will be far less than the errors
> and retries generated from disallowed POSTs. So if we can find a
> volunteer to host this traffic on their website, then we could have
> the redirects go there rather than to openoffice.org or staroffice.de.
> This would not need to be a host volunteering to do this forever. I
> think if we ran it for 2 months we'd get almost everyone, since the
> upgrade checks occur once a week by default.
>
> But we can discuss this more after AOO 3.4.1 is out. No sense
> tempting the OOo 3.1 users to upgrade to AOO 3.4.0 today and then send
> them a notification AOO 3.4.1 a week later. That would be annoying.
Agreed. It may be we can get an Infra VM - ooo-slop. Put a simple apache server with POST and keep the logs for our own analysis. If it crashes only the update process is harmed.
I will close the existing JIRA and then open another explicitly for when we have this decided.
Regards,
Dave
>
> -Rob
>
>>
>>>> There IS an update30 directory there but it isn't actually being used,
>>> and
>>>> is just a dummy file anyway. Maybe we should just delete this one so we
>>>> won't get confused about this one anymore. It was setup in early stages
>>> of
>>>> testing.
>>>>
>>>> Should I just delete ./update30/ProductUpdateService/check.Update -- I
>>> mean
>>>> the whole directory.
>>>>
>>>>
>>>>> -Rob
>>>>>
>>>>>> Regards,
>>>>>> Dave
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>> ----------------------------------------------------------------------------------------
>>>> MzK
>>>>
>>>> "Never express yourself more clearly than you are able to think."
>>>>
>>> --
>>>> Niels Bohr
>>>
>>
>>
>>
>> --
>> ----------------------------------------------------------------------------------------
>> MzK
>>
>> "Never express yourself more clearly than you are able to think."
>> --
>> Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 15, 2012 at 7:31 PM, Kay Schenk <ka...@gmail.com> wrote:
> On Wed, Aug 15, 2012 at 4:04 PM, Rob Weir <ro...@apache.org> wrote:
>
>> On Wed, Aug 15, 2012 at 5:29 PM, Kay Schenk <ka...@gmail.com> wrote:
>> > On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <ro...@apache.org> wrote:
>> >
>> >> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net>
>> >> wrote:
>> >> >
>> >> > On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
>> >> >
>> >> >> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
>> >> >>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <
>> d.s@daniel.shahaf.name>
>> >> wrote:
>> >> >>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21
>> +0200:
>> >> >>>>> Is it possible that somebody from the Apache Infrastructure can
>> >> >>>>> provide a view on which URL the traffic load was soo high that the
>> >> >>>>> servers got in trouble?
>> >> >>>>>
>> >> >>>>
>> >> >>>> POST requests to /ProductUpdateService/check.Update file
>> >> >>>>
>> >> >>>
>> >> >>> For which subdomain, which UpdateXX.openoffice.org ?
>> >> >>
>> >> >> The access log doesn't say, and the error log has
>> >> >>
>> >> >> % fgrep /ProductUpdateService/check.Update error_log | sed -e
>> >> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
>> >> >>
>> >> >> EU:
>> >> >> 232046 update30
>> >> >> 35548 update34
>> >> >> 76543 update35
>> >> >>
>> >> >>
>> >> >> US:
>> >> >> 198996 update30
>> >> >> 33450 update34
>> >> >> 71117 update35
>> >> >> 0 update36
>> >> >
>> >> > We don't see update32 because those do not get redirected in the same
>> >> way because there is no ooo-site/trunk/content/projects/update32
>> >>
>> >
>> > uh oh...this should have been setup before and Oliver said he requested
>> > this in the first post here.
>> >
>> > And you're now saying that all the previous ones have been reverted?
>> >
>> > I think we were OK until this last one, right? update32?
>> >
>> > I think the others should be re-established as they weren't causing
>> > problems, were they?
>> >
>> > The thing is unless we go back to the code for OOo 3.1, we don't know
>> what
>> > it's doing.
>> >
>> > When I asked about this when we had issues for OOo 3.0, I was told it was
>> > fixed in OOo 3.2, so maybe 3.1. has the same issues?
>> >
>> >
>> >
>> >>
>> >> > ./update/aoo341/check.Update
>> >> > ./update/ProductUpdateService/check.Update
>> >> > ./update30/ProductUpdateService/check.Update
>> >> > ./update34/ProductUpdateService/check.Update
>> >> > ./update34/ProductUpdateService/test.Update
>> >> > ./update35/ProductUpdateService/check.Update
>> >> > ./update35/ProductUpdateService/test.Update
>> >> > ./update36/ProductUpdateService/check.Update
>> >> > ./update38/ProductUpdateService/check.Update
>> >> >
>> >> > It looks like 34 and 35 have been trouble, but not as bad as 30.
>> >> >
>> >>
>> >> But haven't 34 and 35 been in production since early July? We've
>> >> certainly seen plenty of downloads triggered by them. They should not
>> >> be giving any errors, since the requests resolve to files on our site.
>> >>
>> >> I wonder, could the errors in those be caused by the outage caused by
>> >> the errors in update30?
>> >>
>> >
>> > Rob...update 30 is completely out of the question, and we simply can not
>> > do it, and reverted it within hours when I first requested it.
>> >
>>
>> What is the issue with update 30? The fact that it does a POST? I
>> don't that would rule it out altogether. But we would need to treat
>> it specially. For example, we could redirect to an isolated server,
>> at Apache or outside, that is able/willing to handle it. If we run it
>> for a month or two we should get the bulk of the upgrades.
>>
>> Or was there some other issue?
>>
>
> The Apache web server, of which AOO is a part, does not allow POSTs so when
> I had infra enable this and redirect the old update30 to the web server, it
> caused MANY errors in a very short period of time (about an hour) and Joe
> reverted it rather quickly. THis was like back in early March or
> something when I was playing around. The update feed itself didn't even DO
> anything but redirect them to a URL (in theory) it was the POST in the code
> for OOo 3.0 that caused all the havoc. When I inquired about this on this
> list, I was told yes, this WAS the case for 3.0 but had been fixed with, I
> thought 3.2.
>
> Anyway, as far as I know, this is the only issue.
>
> I was pretty wary initially about running the feed through the web server
> but was told we should be fine (this after Joe reverted the update30 in
> March) -- and I think we have been for the most part. But, yes, we need
> another box with a web server that would accept POSTs to deal with this --
> 3.0, and it looks like 3.1.
>
OK. That's the impression I had as well -- POST was disabled in
general on ASF servers. We're not going to be able to change that.
But the traffic from permitted POSTs will be far less than the errors
and retries generated from disallowed POSTs. So if we can find a
volunteer to host this traffic on their website, then we could have
the redirects go there rather than to openoffice.org or staroffice.de.
This would not need to be a host volunteering to do this forever. I
think if we ran it for 2 months we'd get almost everyone, since the
upgrade checks occur once a week by default.
But we can discuss this more after AOO 3.4.1 is out. No sense
tempting the OOo 3.1 users to upgrade to AOO 3.4.0 today and then send
them a notification AOO 3.4.1 a week later. That would be annoying.
-Rob
>
>> > There IS an update30 directory there but it isn't actually being used,
>> and
>> > is just a dummy file anyway. Maybe we should just delete this one so we
>> > won't get confused about this one anymore. It was setup in early stages
>> of
>> > testing.
>> >
>> > Should I just delete ./update30/ProductUpdateService/check.Update -- I
>> mean
>> > the whole directory.
>> >
>> >
>> >> -Rob
>> >>
>> >> > Regards,
>> >> > Dave
>> >>
>> >
>> >
>> >
>> > --
>> >
>> ----------------------------------------------------------------------------------------
>> > MzK
>> >
>> > "Never express yourself more clearly than you are able to think."
>> >
>> --
>> > Niels Bohr
>>
>
>
>
> --
> ----------------------------------------------------------------------------------------
> MzK
>
> "Never express yourself more clearly than you are able to think."
> --
> Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Kay Schenk <ka...@gmail.com>.
On Wed, Aug 15, 2012 at 4:04 PM, Rob Weir <ro...@apache.org> wrote:
> On Wed, Aug 15, 2012 at 5:29 PM, Kay Schenk <ka...@gmail.com> wrote:
> > On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <ro...@apache.org> wrote:
> >
> >> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net>
> >> wrote:
> >> >
> >> > On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
> >> >
> >> >> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
> >> >>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <
> d.s@daniel.shahaf.name>
> >> wrote:
> >> >>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21
> +0200:
> >> >>>>> Is it possible that somebody from the Apache Infrastructure can
> >> >>>>> provide a view on which URL the traffic load was soo high that the
> >> >>>>> servers got in trouble?
> >> >>>>>
> >> >>>>
> >> >>>> POST requests to /ProductUpdateService/check.Update file
> >> >>>>
> >> >>>
> >> >>> For which subdomain, which UpdateXX.openoffice.org ?
> >> >>
> >> >> The access log doesn't say, and the error log has
> >> >>
> >> >> % fgrep /ProductUpdateService/check.Update error_log | sed -e
> >> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
> >> >>
> >> >> EU:
> >> >> 232046 update30
> >> >> 35548 update34
> >> >> 76543 update35
> >> >>
> >> >>
> >> >> US:
> >> >> 198996 update30
> >> >> 33450 update34
> >> >> 71117 update35
> >> >> 0 update36
> >> >
> >> > We don't see update32 because those do not get redirected in the same
> >> way because there is no ooo-site/trunk/content/projects/update32
> >>
> >
> > uh oh...this should have been setup before and Oliver said he requested
> > this in the first post here.
> >
> > And you're now saying that all the previous ones have been reverted?
> >
> > I think we were OK until this last one, right? update32?
> >
> > I think the others should be re-established as they weren't causing
> > problems, were they?
> >
> > The thing is unless we go back to the code for OOo 3.1, we don't know
> what
> > it's doing.
> >
> > When I asked about this when we had issues for OOo 3.0, I was told it was
> > fixed in OOo 3.2, so maybe 3.1. has the same issues?
> >
> >
> >
> >>
> >> > ./update/aoo341/check.Update
> >> > ./update/ProductUpdateService/check.Update
> >> > ./update30/ProductUpdateService/check.Update
> >> > ./update34/ProductUpdateService/check.Update
> >> > ./update34/ProductUpdateService/test.Update
> >> > ./update35/ProductUpdateService/check.Update
> >> > ./update35/ProductUpdateService/test.Update
> >> > ./update36/ProductUpdateService/check.Update
> >> > ./update38/ProductUpdateService/check.Update
> >> >
> >> > It looks like 34 and 35 have been trouble, but not as bad as 30.
> >> >
> >>
> >> But haven't 34 and 35 been in production since early July? We've
> >> certainly seen plenty of downloads triggered by them. They should not
> >> be giving any errors, since the requests resolve to files on our site.
> >>
> >> I wonder, could the errors in those be caused by the outage caused by
> >> the errors in update30?
> >>
> >
> > Rob...update 30 is completely out of the question, and we simply can not
> > do it, and reverted it within hours when I first requested it.
> >
>
> What is the issue with update 30? The fact that it does a POST? I
> don't that would rule it out altogether. But we would need to treat
> it specially. For example, we could redirect to an isolated server,
> at Apache or outside, that is able/willing to handle it. If we run it
> for a month or two we should get the bulk of the upgrades.
>
> Or was there some other issue?
>
The Apache web server, of which AOO is a part, does not allow POSTs so when
I had infra enable this and redirect the old update30 to the web server, it
caused MANY errors in a very short period of time (about an hour) and Joe
reverted it rather quickly. THis was like back in early March or
something when I was playing around. The update feed itself didn't even DO
anything but redirect them to a URL (in theory) it was the POST in the code
for OOo 3.0 that caused all the havoc. When I inquired about this on this
list, I was told yes, this WAS the case for 3.0 but had been fixed with, I
thought 3.2.
Anyway, as far as I know, this is the only issue.
I was pretty wary initially about running the feed through the web server
but was told we should be fine (this after Joe reverted the update30 in
March) -- and I think we have been for the most part. But, yes, we need
another box with a web server that would accept POSTs to deal with this --
3.0, and it looks like 3.1.
> > There IS an update30 directory there but it isn't actually being used,
> and
> > is just a dummy file anyway. Maybe we should just delete this one so we
> > won't get confused about this one anymore. It was setup in early stages
> of
> > testing.
> >
> > Should I just delete ./update30/ProductUpdateService/check.Update -- I
> mean
> > the whole directory.
> >
> >
> >> -Rob
> >>
> >> > Regards,
> >> > Dave
> >>
> >
> >
> >
> > --
> >
> ----------------------------------------------------------------------------------------
> > MzK
> >
> > "Never express yourself more clearly than you are able to think."
> >
> --
> > Niels Bohr
>
--
----------------------------------------------------------------------------------------
MzK
"Never express yourself more clearly than you are able to think."
--
Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 15, 2012 at 5:29 PM, Kay Schenk <ka...@gmail.com> wrote:
> On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <ro...@apache.org> wrote:
>
>> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net>
>> wrote:
>> >
>> > On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
>> >
>> >> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
>> >>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <d....@daniel.shahaf.name>
>> wrote:
>> >>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>> >>>>> Is it possible that somebody from the Apache Infrastructure can
>> >>>>> provide a view on which URL the traffic load was soo high that the
>> >>>>> servers got in trouble?
>> >>>>>
>> >>>>
>> >>>> POST requests to /ProductUpdateService/check.Update file
>> >>>>
>> >>>
>> >>> For which subdomain, which UpdateXX.openoffice.org ?
>> >>
>> >> The access log doesn't say, and the error log has
>> >>
>> >> % fgrep /ProductUpdateService/check.Update error_log | sed -e
>> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
>> >>
>> >> EU:
>> >> 232046 update30
>> >> 35548 update34
>> >> 76543 update35
>> >>
>> >>
>> >> US:
>> >> 198996 update30
>> >> 33450 update34
>> >> 71117 update35
>> >> 0 update36
>> >
>> > We don't see update32 because those do not get redirected in the same
>> way because there is no ooo-site/trunk/content/projects/update32
>>
>
> uh oh...this should have been setup before and Oliver said he requested
> this in the first post here.
>
> And you're now saying that all the previous ones have been reverted?
>
> I think we were OK until this last one, right? update32?
>
> I think the others should be re-established as they weren't causing
> problems, were they?
>
> The thing is unless we go back to the code for OOo 3.1, we don't know what
> it's doing.
>
> When I asked about this when we had issues for OOo 3.0, I was told it was
> fixed in OOo 3.2, so maybe 3.1. has the same issues?
>
>
>
>>
>> > ./update/aoo341/check.Update
>> > ./update/ProductUpdateService/check.Update
>> > ./update30/ProductUpdateService/check.Update
>> > ./update34/ProductUpdateService/check.Update
>> > ./update34/ProductUpdateService/test.Update
>> > ./update35/ProductUpdateService/check.Update
>> > ./update35/ProductUpdateService/test.Update
>> > ./update36/ProductUpdateService/check.Update
>> > ./update38/ProductUpdateService/check.Update
>> >
>> > It looks like 34 and 35 have been trouble, but not as bad as 30.
>> >
>>
>> But haven't 34 and 35 been in production since early July? We've
>> certainly seen plenty of downloads triggered by them. They should not
>> be giving any errors, since the requests resolve to files on our site.
>>
>> I wonder, could the errors in those be caused by the outage caused by
>> the errors in update30?
>>
>
> Rob...update 30 is completely out of the question, and we simply can not
> do it, and reverted it within hours when I first requested it.
>
What is the issue with update 30? The fact that it does a POST? I
don't that would rule it out altogether. But we would need to treat
it specially. For example, we could redirect to an isolated server,
at Apache or outside, that is able/willing to handle it. If we run it
for a month or two we should get the bulk of the upgrades.
Or was there some other issue?
> There IS an update30 directory there but it isn't actually being used, and
> is just a dummy file anyway. Maybe we should just delete this one so we
> won't get confused about this one anymore. It was setup in early stages of
> testing.
>
> Should I just delete ./update30/ProductUpdateService/check.Update -- I mean
> the whole directory.
>
>
>> -Rob
>>
>> > Regards,
>> > Dave
>>
>
>
>
> --
> ----------------------------------------------------------------------------------------
> MzK
>
> "Never express yourself more clearly than you are able to think."
> --
> Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Kay Schenk <ka...@gmail.com>.
On Wed, Aug 15, 2012 at 3:01 PM, Kay Schenk <ka...@gmail.com> wrote:
>
>
> On Wed, Aug 15, 2012 at 2:49 PM, Dave Fisher <da...@comcast.net>wrote:
>
>>
>> On Aug 15, 2012, at 2:29 PM, Kay Schenk wrote:
>>
>> > On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <ro...@apache.org> wrote:
>> >
>> >> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net>
>> >> wrote:
>> >>>
>> >>> On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
>> >>>
>> >>>> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
>> >>>>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <
>> d.s@daniel.shahaf.name>
>> >> wrote:
>> >>>>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21
>> +0200:
>> >>>>>>> Is it possible that somebody from the Apache Infrastructure can
>> >>>>>>> provide a view on which URL the traffic load was soo high that the
>> >>>>>>> servers got in trouble?
>> >>>>>>>
>> >>>>>>
>> >>>>>> POST requests to /ProductUpdateService/check.Update file
>> >>>>>>
>> >>>>>
>> >>>>> For which subdomain, which UpdateXX.openoffice.org ?
>> >>>>
>> >>>> The access log doesn't say, and the error log has
>> >>>>
>> >>>> % fgrep /ProductUpdateService/check.Update error_log | sed -e
>> >> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
>> >>>>
>> >>>> EU:
>> >>>> 232046 update30
>> >>>> 35548 update34
>> >>>> 76543 update35
>> >>>>
>> >>>>
>> >>>> US:
>> >>>> 198996 update30
>> >>>> 33450 update34
>> >>>> 71117 update35
>> >>>> 0 update36
>> >>>
>> >>> We don't see update32 because those do not get redirected in the same
>> >> way because there is no ooo-site/trunk/content/projects/update32
>> >>
>> >
>> > uh oh...this should have been setup before and Oliver said he requested
>> > this in the first post here.
>> >
>> > And you're now saying that all the previous ones have been reverted?
>>
>> The zone file is now:
>>
>> ; update.services CNAME www.openoffice.org.
>>
>>
>>
>>
>> ; WARNING WARNING WARNING
>> ; Changing the above entries to point to eos can overload it. Do not
>> enable
>> ; them unless either eos is prepared for the load or the TTL is suitably
>> ; lowered
>> update.services CNAME sd-web4.staroffice.de.
>> update23.services CNAME sd-web2.staroffice.de.
>> update24.services CNAME sd-web2.staroffice.de.
>> update30.services CNAME sd-web2.staroffice.de.
>> update31.services CNAME sd-web2.staroffice.de.
>> update32.services CNAME sd-web2.staroffice.de.
>> update33.services CNAME sd-web2.staroffice.de.
>>
>>
>>
>>
>> update34.services CNAME www.openoffice.org.
>> update35.services CNAME www.openoffice.org.
>> update36.services CNAME www.openoffice.org.
>> update38.services CNAME www.openoffice.org.
>>
>>
>> >
>> > I think we were OK until this last one, right? update32?
>>
>> Only yesterday's changes to DNS were reverted.
>>
>
> OK, good...it seems we can't go below update34 -- used for OO 3.2 without
> further investigation.
>
> Meanwhile I will delete the
>
> ./update30/ProductUpdateService/check.Update
>
> so it causes no further confusion...
>
ok, this is gone now...the ones that remain with numbers following updatexx
are in use.
/projects/updatexx/....
as well as
/projects/update/ which is used for aoo34
>
>
>> >
>> > I think the others should be re-established as they weren't causing
>> > problems, were they?
>> >
>> > The thing is unless we go back to the code for OOo 3.1, we don't know
>> what
>> > it's doing.
>> >
>> > When I asked about this when we had issues for OOo 3.0, I was told it
>> was
>> > fixed in OOo 3.2, so maybe 3.1. has the same issues?
>> >
>> >
>> >
>> >>
>> >>> ./update/aoo341/check.Update
>> >>> ./update/ProductUpdateService/check.Update
>> >>> ./update30/ProductUpdateService/check.Update
>> >>> ./update34/ProductUpdateService/check.Update
>> >>> ./update34/ProductUpdateService/test.Update
>> >>> ./update35/ProductUpdateService/check.Update
>> >>> ./update35/ProductUpdateService/test.Update
>> >>> ./update36/ProductUpdateService/check.Update
>> >>> ./update38/ProductUpdateService/check.Update
>> >>>
>> >>> It looks like 34 and 35 have been trouble, but not as bad as 30.
>> >>>
>> >>
>> >> But haven't 34 and 35 been in production since early July? We've
>> >> certainly seen plenty of downloads triggered by them. They should not
>> >> be giving any errors, since the requests resolve to files on our site.
>> >>
>> >> I wonder, could the errors in those be caused by the outage caused by
>> >> the errors in update30?
>> >>
>> >
>> > Rob...update 30 is completely out of the question, and we simply can
>> not
>> > do it, and reverted it within hours when I first requested it.
>> >
>> > There IS an update30 directory there but it isn't actually being used,
>> and
>> > is just a dummy file anyway. Maybe we should just delete this one so we
>> > won't get confused about this one anymore. It was setup in early stages
>> of
>> > testing.
>> >
>> > Should I just delete ./update30/ProductUpdateService/check.Update -- I
>> mean
>> > the whole directory.
>> >
>> >
>> >> -Rob
>> >>
>> >>> Regards,
>> >>> Dave
>> >>
>> >
>> >
>> >
>> > --
>> >
>> ----------------------------------------------------------------------------------------
>> > MzK
>> >
>> > "Never express yourself more clearly than you are able to think."
>> >
>> --
>> > Niels Bohr
>>
>>
>
>
> --
>
> ----------------------------------------------------------------------------------------
> MzK
>
> "Never express yourself more clearly than you are able to think."
> --
> Niels Bohr
>
--
----------------------------------------------------------------------------------------
MzK
"Never express yourself more clearly than you are able to think."
--
Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Kay Schenk <ka...@gmail.com>.
On Wed, Aug 15, 2012 at 2:49 PM, Dave Fisher <da...@comcast.net> wrote:
>
> On Aug 15, 2012, at 2:29 PM, Kay Schenk wrote:
>
> > On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <ro...@apache.org> wrote:
> >
> >> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net>
> >> wrote:
> >>>
> >>> On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
> >>>
> >>>> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
> >>>>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <
> d.s@daniel.shahaf.name>
> >> wrote:
> >>>>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
> >>>>>>> Is it possible that somebody from the Apache Infrastructure can
> >>>>>>> provide a view on which URL the traffic load was soo high that the
> >>>>>>> servers got in trouble?
> >>>>>>>
> >>>>>>
> >>>>>> POST requests to /ProductUpdateService/check.Update file
> >>>>>>
> >>>>>
> >>>>> For which subdomain, which UpdateXX.openoffice.org ?
> >>>>
> >>>> The access log doesn't say, and the error log has
> >>>>
> >>>> % fgrep /ProductUpdateService/check.Update error_log | sed -e
> >> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
> >>>>
> >>>> EU:
> >>>> 232046 update30
> >>>> 35548 update34
> >>>> 76543 update35
> >>>>
> >>>>
> >>>> US:
> >>>> 198996 update30
> >>>> 33450 update34
> >>>> 71117 update35
> >>>> 0 update36
> >>>
> >>> We don't see update32 because those do not get redirected in the same
> >> way because there is no ooo-site/trunk/content/projects/update32
> >>
> >
> > uh oh...this should have been setup before and Oliver said he requested
> > this in the first post here.
> >
> > And you're now saying that all the previous ones have been reverted?
>
> The zone file is now:
>
> ; update.services CNAME www.openoffice.org.
>
>
>
>
> ; WARNING WARNING WARNING
> ; Changing the above entries to point to eos can overload it. Do not
> enable
> ; them unless either eos is prepared for the load or the TTL is suitably
> ; lowered
> update.services CNAME sd-web4.staroffice.de.
> update23.services CNAME sd-web2.staroffice.de.
> update24.services CNAME sd-web2.staroffice.de.
> update30.services CNAME sd-web2.staroffice.de.
> update31.services CNAME sd-web2.staroffice.de.
> update32.services CNAME sd-web2.staroffice.de.
> update33.services CNAME sd-web2.staroffice.de.
>
>
>
>
> update34.services CNAME www.openoffice.org.
> update35.services CNAME www.openoffice.org.
> update36.services CNAME www.openoffice.org.
> update38.services CNAME www.openoffice.org.
>
>
> >
> > I think we were OK until this last one, right? update32?
>
> Only yesterday's changes to DNS were reverted.
>
OK, good...it seems we can't go below update34 -- used for OO 3.2 without
further investigation.
Meanwhile I will delete the
./update30/ProductUpdateService/check.Update
so it causes no further confusion...
> >
> > I think the others should be re-established as they weren't causing
> > problems, were they?
> >
> > The thing is unless we go back to the code for OOo 3.1, we don't know
> what
> > it's doing.
> >
> > When I asked about this when we had issues for OOo 3.0, I was told it was
> > fixed in OOo 3.2, so maybe 3.1. has the same issues?
> >
> >
> >
> >>
> >>> ./update/aoo341/check.Update
> >>> ./update/ProductUpdateService/check.Update
> >>> ./update30/ProductUpdateService/check.Update
> >>> ./update34/ProductUpdateService/check.Update
> >>> ./update34/ProductUpdateService/test.Update
> >>> ./update35/ProductUpdateService/check.Update
> >>> ./update35/ProductUpdateService/test.Update
> >>> ./update36/ProductUpdateService/check.Update
> >>> ./update38/ProductUpdateService/check.Update
> >>>
> >>> It looks like 34 and 35 have been trouble, but not as bad as 30.
> >>>
> >>
> >> But haven't 34 and 35 been in production since early July? We've
> >> certainly seen plenty of downloads triggered by them. They should not
> >> be giving any errors, since the requests resolve to files on our site.
> >>
> >> I wonder, could the errors in those be caused by the outage caused by
> >> the errors in update30?
> >>
> >
> > Rob...update 30 is completely out of the question, and we simply can not
> > do it, and reverted it within hours when I first requested it.
> >
> > There IS an update30 directory there but it isn't actually being used,
> and
> > is just a dummy file anyway. Maybe we should just delete this one so we
> > won't get confused about this one anymore. It was setup in early stages
> of
> > testing.
> >
> > Should I just delete ./update30/ProductUpdateService/check.Update -- I
> mean
> > the whole directory.
> >
> >
> >> -Rob
> >>
> >>> Regards,
> >>> Dave
> >>
> >
> >
> >
> > --
> >
> ----------------------------------------------------------------------------------------
> > MzK
> >
> > "Never express yourself more clearly than you are able to think."
> > --
> > Niels Bohr
>
>
--
----------------------------------------------------------------------------------------
MzK
"Never express yourself more clearly than you are able to think."
--
Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Dave Fisher <da...@comcast.net>.
On Aug 15, 2012, at 2:29 PM, Kay Schenk wrote:
> On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <ro...@apache.org> wrote:
>
>> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net>
>> wrote:
>>>
>>> On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
>>>
>>>> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
>>>>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <d....@daniel.shahaf.name>
>> wrote:
>>>>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>>>>>>> Is it possible that somebody from the Apache Infrastructure can
>>>>>>> provide a view on which URL the traffic load was soo high that the
>>>>>>> servers got in trouble?
>>>>>>>
>>>>>>
>>>>>> POST requests to /ProductUpdateService/check.Update file
>>>>>>
>>>>>
>>>>> For which subdomain, which UpdateXX.openoffice.org ?
>>>>
>>>> The access log doesn't say, and the error log has
>>>>
>>>> % fgrep /ProductUpdateService/check.Update error_log | sed -e
>> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
>>>>
>>>> EU:
>>>> 232046 update30
>>>> 35548 update34
>>>> 76543 update35
>>>>
>>>>
>>>> US:
>>>> 198996 update30
>>>> 33450 update34
>>>> 71117 update35
>>>> 0 update36
>>>
>>> We don't see update32 because those do not get redirected in the same
>> way because there is no ooo-site/trunk/content/projects/update32
>>
>
> uh oh...this should have been setup before and Oliver said he requested
> this in the first post here.
>
> And you're now saying that all the previous ones have been reverted?
The zone file is now:
; update.services CNAME www.openoffice.org.
; WARNING WARNING WARNING
; Changing the above entries to point to eos can overload it. Do not enable
; them unless either eos is prepared for the load or the TTL is suitably
; lowered
update.services CNAME sd-web4.staroffice.de.
update23.services CNAME sd-web2.staroffice.de.
update24.services CNAME sd-web2.staroffice.de.
update30.services CNAME sd-web2.staroffice.de.
update31.services CNAME sd-web2.staroffice.de.
update32.services CNAME sd-web2.staroffice.de.
update33.services CNAME sd-web2.staroffice.de.
update34.services CNAME www.openoffice.org.
update35.services CNAME www.openoffice.org.
update36.services CNAME www.openoffice.org.
update38.services CNAME www.openoffice.org.
>
> I think we were OK until this last one, right? update32?
Only yesterday's changes to DNS were reverted.
>
> I think the others should be re-established as they weren't causing
> problems, were they?
>
> The thing is unless we go back to the code for OOo 3.1, we don't know what
> it's doing.
>
> When I asked about this when we had issues for OOo 3.0, I was told it was
> fixed in OOo 3.2, so maybe 3.1. has the same issues?
>
>
>
>>
>>> ./update/aoo341/check.Update
>>> ./update/ProductUpdateService/check.Update
>>> ./update30/ProductUpdateService/check.Update
>>> ./update34/ProductUpdateService/check.Update
>>> ./update34/ProductUpdateService/test.Update
>>> ./update35/ProductUpdateService/check.Update
>>> ./update35/ProductUpdateService/test.Update
>>> ./update36/ProductUpdateService/check.Update
>>> ./update38/ProductUpdateService/check.Update
>>>
>>> It looks like 34 and 35 have been trouble, but not as bad as 30.
>>>
>>
>> But haven't 34 and 35 been in production since early July? We've
>> certainly seen plenty of downloads triggered by them. They should not
>> be giving any errors, since the requests resolve to files on our site.
>>
>> I wonder, could the errors in those be caused by the outage caused by
>> the errors in update30?
>>
>
> Rob...update 30 is completely out of the question, and we simply can not
> do it, and reverted it within hours when I first requested it.
>
> There IS an update30 directory there but it isn't actually being used, and
> is just a dummy file anyway. Maybe we should just delete this one so we
> won't get confused about this one anymore. It was setup in early stages of
> testing.
>
> Should I just delete ./update30/ProductUpdateService/check.Update -- I mean
> the whole directory.
>
>
>> -Rob
>>
>>> Regards,
>>> Dave
>>
>
>
>
> --
> ----------------------------------------------------------------------------------------
> MzK
>
> "Never express yourself more clearly than you are able to think."
> --
> Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Kay Schenk <ka...@gmail.com>.
On Wed, Aug 15, 2012 at 11:15 AM, Rob Weir <ro...@apache.org> wrote:
> On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net>
> wrote:
> >
> > On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
> >
> >> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
> >>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <d....@daniel.shahaf.name>
> wrote:
> >>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
> >>>>> Is it possible that somebody from the Apache Infrastructure can
> >>>>> provide a view on which URL the traffic load was soo high that the
> >>>>> servers got in trouble?
> >>>>>
> >>>>
> >>>> POST requests to /ProductUpdateService/check.Update file
> >>>>
> >>>
> >>> For which subdomain, which UpdateXX.openoffice.org ?
> >>
> >> The access log doesn't say, and the error log has
> >>
> >> % fgrep /ProductUpdateService/check.Update error_log | sed -e
> 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
> >>
> >> EU:
> >> 232046 update30
> >> 35548 update34
> >> 76543 update35
> >>
> >>
> >> US:
> >> 198996 update30
> >> 33450 update34
> >> 71117 update35
> >> 0 update36
> >
> > We don't see update32 because those do not get redirected in the same
> way because there is no ooo-site/trunk/content/projects/update32
>
uh oh...this should have been setup before and Oliver said he requested
this in the first post here.
And you're now saying that all the previous ones have been reverted?
I think we were OK until this last one, right? update32?
I think the others should be re-established as they weren't causing
problems, were they?
The thing is unless we go back to the code for OOo 3.1, we don't know what
it's doing.
When I asked about this when we had issues for OOo 3.0, I was told it was
fixed in OOo 3.2, so maybe 3.1. has the same issues?
>
> > ./update/aoo341/check.Update
> > ./update/ProductUpdateService/check.Update
> > ./update30/ProductUpdateService/check.Update
> > ./update34/ProductUpdateService/check.Update
> > ./update34/ProductUpdateService/test.Update
> > ./update35/ProductUpdateService/check.Update
> > ./update35/ProductUpdateService/test.Update
> > ./update36/ProductUpdateService/check.Update
> > ./update38/ProductUpdateService/check.Update
> >
> > It looks like 34 and 35 have been trouble, but not as bad as 30.
> >
>
> But haven't 34 and 35 been in production since early July? We've
> certainly seen plenty of downloads triggered by them. They should not
> be giving any errors, since the requests resolve to files on our site.
>
> I wonder, could the errors in those be caused by the outage caused by
> the errors in update30?
>
Rob...update 30 is completely out of the question, and we simply can not
do it, and reverted it within hours when I first requested it.
There IS an update30 directory there but it isn't actually being used, and
is just a dummy file anyway. Maybe we should just delete this one so we
won't get confused about this one anymore. It was setup in early stages of
testing.
Should I just delete ./update30/ProductUpdateService/check.Update -- I mean
the whole directory.
> -Rob
>
> > Regards,
> > Dave
>
--
----------------------------------------------------------------------------------------
MzK
"Never express yourself more clearly than you are able to think."
--
Niels Bohr
Re: Registration and Update Services - What Will Be The Load?
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 15, 2012 at 2:05 PM, Dave Fisher <da...@comcast.net> wrote:
>
> On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
>
>> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
>>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
>>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>>>>> Is it possible that somebody from the Apache Infrastructure can
>>>>> provide a view on which URL the traffic load was soo high that the
>>>>> servers got in trouble?
>>>>>
>>>>
>>>> POST requests to /ProductUpdateService/check.Update file
>>>>
>>>
>>> For which subdomain, which UpdateXX.openoffice.org ?
>>
>> The access log doesn't say, and the error log has
>>
>> % fgrep /ProductUpdateService/check.Update error_log | sed -e 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
>>
>> EU:
>> 232046 update30
>> 35548 update34
>> 76543 update35
>>
>>
>> US:
>> 198996 update30
>> 33450 update34
>> 71117 update35
>> 0 update36
>
> We don't see update32 because those do not get redirected in the same way because there is no ooo-site/trunk/content/projects/update32
>
> ./update/aoo341/check.Update
> ./update/ProductUpdateService/check.Update
> ./update30/ProductUpdateService/check.Update
> ./update34/ProductUpdateService/check.Update
> ./update34/ProductUpdateService/test.Update
> ./update35/ProductUpdateService/check.Update
> ./update35/ProductUpdateService/test.Update
> ./update36/ProductUpdateService/check.Update
> ./update38/ProductUpdateService/check.Update
>
> It looks like 34 and 35 have been trouble, but not as bad as 30.
>
But haven't 34 and 35 been in production since early July? We've
certainly seen plenty of downloads triggered by them. They should not
be giving any errors, since the requests resolve to files on our site.
I wonder, could the errors in those be caused by the outage caused by
the errors in update30?
-Rob
> Regards,
> Dave
Re: Registration and Update Services - What Will Be The Load?
Posted by Dave Fisher <da...@comcast.net>.
On Aug 15, 2012, at 10:28 AM, Daniel Shahaf wrote:
> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>>>> Is it possible that somebody from the Apache Infrastructure can
>>>> provide a view on which URL the traffic load was soo high that the
>>>> servers got in trouble?
>>>>
>>>
>>> POST requests to /ProductUpdateService/check.Update file
>>>
>>
>> For which subdomain, which UpdateXX.openoffice.org ?
>
> The access log doesn't say, and the error log has
>
> % fgrep /ProductUpdateService/check.Update error_log | sed -e 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
>
> EU:
> 232046 update30
> 35548 update34
> 76543 update35
>
>
> US:
> 198996 update30
> 33450 update34
> 71117 update35
> 0 update36
We don't see update32 because those do not get redirected in the same way because there is no ooo-site/trunk/content/projects/update32
./update/aoo341/check.Update
./update/ProductUpdateService/check.Update
./update30/ProductUpdateService/check.Update
./update34/ProductUpdateService/check.Update
./update34/ProductUpdateService/test.Update
./update35/ProductUpdateService/check.Update
./update35/ProductUpdateService/test.Update
./update36/ProductUpdateService/check.Update
./update38/ProductUpdateService/check.Update
It looks like 34 and 35 have been trouble, but not as bad as 30.
Regards,
Dave
Re: Registration and Update Services - What Will Be The Load?
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi Daniel,
On 15.08.2012 19:28, Daniel Shahaf wrote:
> Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
>> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
>>> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>>>> Is it possible that somebody from the Apache Infrastructure can
>>>> provide a view on which URL the traffic load was soo high that the
>>>> servers got in trouble?
>>>>
>>>
>>> POST requests to /ProductUpdateService/check.Update file
>>>
>>
>> For which subdomain, which UpdateXX.openoffice.org ?
>
> The access log doesn't say, and the error log has
>
> % fgrep /ProductUpdateService/check.Update error_log | sed -e 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
>
> EU:
> 232046 update30
> 35548 update34
> 76543 update35
>
>
> US:
> 198996 update30
> 33450 update34
> 71117 update35
> 0 update36
>
>
Currently, I am not sure about the changes to the
infrastructure/dns/zones/openoffice.org file
You mentioned the revisions r828963 and r828965, but I did not find the
corresponding SVN repository
Can you please point me to the repository or provide the change set?
Thank you in advance,
Oliver.
Re: Registration and Update Services - What Will Be The Load?
Posted by Daniel Shahaf <da...@apache.org>.
Rob Weir wrote on Wed, Aug 15, 2012 at 13:11:43 -0400:
> On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> > Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
> >> Is it possible that somebody from the Apache Infrastructure can
> >> provide a view on which URL the traffic load was soo high that the
> >> servers got in trouble?
> >>
> >
> > POST requests to /ProductUpdateService/check.Update file
> >
>
> For which subdomain, which UpdateXX.openoffice.org ?
The access log doesn't say, and the error log has
% fgrep /ProductUpdateService/check.Update error_log | sed -e 's#^.*/content/projects/##' | cut -d/ -f1 | sort | uniq -c
EU:
232046 update30
35548 update34
76543 update35
US:
198996 update30
33450 update34
71117 update35
0 update36
Re: Registration and Update Services - What Will Be The Load?
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 15, 2012 at 1:05 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Oliver-Rainer Wittmann wrote on Wed, Aug 15, 2012 at 16:32:21 +0200:
>> Is it possible that somebody from the Apache Infrastructure can
>> provide a view on which URL the traffic load was soo high that the
>> servers got in trouble?
>>
>
> POST requests to /ProductUpdateService/check.Update file
>
For which subdomain, which UpdateXX.openoffice.org ?
-Rob
>> Best regards, Oliver.
Re: Registration and Update Services - What Will Be The Load?
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 15.08.2012 16:00, Dave Fisher wrote:
> Hi,
>
> Apologies to the Apache Infra team, the load caused by implementing
> INFRA-5112 caused trouble.
>
> The changes to update*.services.openoffice.org to point to www.openoffice.org
> was reverted.
>
> This was due to added load on the Apache Infrastructure.
>
> Before we can proceed with INFRA-5112 and INFRA-5144 we absolutely must have
> some estimates about the volume of requests that will be received.
>
When I saw the problems on www.openoffice.org in the morning I already suggested
that this might be related to the established redirects. I reported my
assumption at #asfinfra in the morning.
If I remembering it correct, Kay S. and Joe S. already observed something like
this earlier this year.
Unfortunately, I can not provide any volume.
In the past more than 100 million download of OpenOffice.org package had been
counted by the OpenOffice.org community. I do not know how much OOo installation
are really active and how the distribution between the different versions are.
What I know is the following:
(1) update38.services.openoffice.org is used by OOo 3.4 Beta and released AOO
3.4 - a redirect for it has been established at 2012-05-21 and the traffic can
be handled.
(2) update36.services.openoffice.org is used by OOo 3.3 - a redirect for it has
been established at 2012-06-04 and the traffic can be handled.
(3) update35|34.services.openoffice.org is used by OOo 3.2.1 and OOo 3.2 -
redirects for then have been established at 2012-07-12 and the traffic can be
handled.
(4) update33.services.openoffice.org is _not_ used - a redirect is _not_
necessary. Does occur any traffic on this URL?
(5) update32.services.openoffice.org is used by OOo 3.1.1 and OOo 3.1 - a
redirect has been requested and was established today. Due to the server load it
has been reverted. Is the traffic data available?
(6) update31.services.openoffice.org is _not_ used - a redirect is _not_
necessary. Does occur any traffic on this URL?
(7) update30.services.openoffice.org is used by OOo 3.0.1 and OOo 3.0. Does
occur any traffic on this URL?
(8) update.services.openoffice.org seems to be used by OOo 2.x version (at least
my test installation of OOo 2.2 uses it). Is the traffic data available? When I
remember it correct Kay S. and Joe S. observed the above mentioned problems
earlier this year with this URL.
Is it possible that somebody from the Apache Infrastructure can provide a view
on which URL the traffic load was soo high that the servers got in trouble?
Best regards, Oliver.