You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whimsical.apache.org by "Roy T. Fielding" <fi...@gbiv.com> on 2020/10/02 18:43:33 UTC
agenda websocket seems to be down
I was going to send out reminders under "missing" but the agenda
won't fill the email form after clicking on "send initial reminders".
My web console shows 500 errors every few minutes
GET wss://whimsy.apache.org/board/agenda/websocket/[HTTP/1.1 500 Internal Server Error 763ms]
GET
wss://whimsy.apache.org/board/agenda/websocket/
Status
500
Internal Server Error
VersionHTTP/1.1
Transferred253 B (0 B size)
Accept-Ranges
bytes
Connection
close
Content-Length
5220
Content-Type
text/html
Date
Fri, 02 Oct 2020 18:37:16 GMT
I would guess this is the issue that Sam talked about earlier,
but I thought it was fixed already.
Er, looks like the error logs just rotated. Is some work being done?
....Roy
Re: agenda websocket seems to be down
Posted by Niclas Hedhman <ni...@apache.org>.
Since I can't see any "Approved" on any reports, I assume that my previous report about "problems" about Board Agenda Tool on board@ may in fact be not only for myself. And if so, it will get chaotic in the near future...
1. I am seeing websocket problems in the Console, but not all the time. I don't think it is a primary problem. When they appear they contain a 503.
2. I have seen that "Bootstrap requires jquery.js" in the log sometimes. Although I don't think that is a primary problem, it could indicate that "loading order" in web pages are an issue.
3. It happens that after the "hang" comes into play, that the https://whimsy.apache.org main page doesn't fully load (i.e. the "loading icon in the title tab is indicating processing"). 7 resources are GET'd; /, bootstrap-min.css, jquery-min.js, bootstrap-min.js, asf-logo.png, whimsy.svg, favicon.ico, which seems correct, and the HTML is fully loaded. So, what could it possibly be that the page isn't "done"
4. When the serious hang (3.) and I manually go to https://whimsy.apache.org/board/agenda/, there is no response at all from the server. Nothing, not a byte.
// Niclas
On 2020/10/02 18:43:33, "Roy T. Fielding" <fi...@gbiv.com> wrote:
> I was going to send out reminders under "missing" but the agenda
> won't fill the email form after clicking on "send initial reminders".
>
> My web console shows 500 errors every few minutes
>
> GET wss://whimsy.apache.org/board/agenda/websocket/[HTTP/1.1 500 Internal Server Error 763ms]
>
> GET
> wss://whimsy.apache.org/board/agenda/websocket/
> Status
> 500
> Internal Server Error
> VersionHTTP/1.1
> Transferred253 B (0 B size)
> Accept-Ranges
> bytes
> Connection
> close
> Content-Length
> 5220
> Content-Type
> text/html
> Date
> Fri, 02 Oct 2020 18:37:16 GMT
>
> I would guess this is the issue that Sam talked about earlier,
> but I thought it was fixed already.
>
> Er, looks like the error logs just rotated. Is some work being done?
>
> ....Roy
>
>
Re: agenda websocket seems to be down
Posted by sebb <se...@gmail.com>.
Well done!
On Sun, 25 Oct 2020 at 01:27, Sam Ruby <ru...@intertwingly.net> wrote:
>
> On Tue, Oct 6, 2020 at 4:53 PM Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > When the client
> > connected to the websocket process directly it made a lot of sense for
> > that process to implement SSL. Now that that is no longer the case,
> > it no longer is necessary. In fact, it no longer is necessary for the
> > websocket to be running as root, as that is only done to enable it to
> > access the certificates.
>
> Dropping the internal SSL connection enables browsers (including
> Firefox) to connect to the websocket again.
>
> - Sam Ruby
Re: agenda websocket seems to be down
Posted by Sam Ruby <ru...@intertwingly.net>.
On Tue, Oct 6, 2020 at 4:53 PM Sam Ruby <ru...@intertwingly.net> wrote:
>
> When the client
> connected to the websocket process directly it made a lot of sense for
> that process to implement SSL. Now that that is no longer the case,
> it no longer is necessary. In fact, it no longer is necessary for the
> websocket to be running as root, as that is only done to enable it to
> access the certificates.
Dropping the internal SSL connection enables browsers (including
Firefox) to connect to the websocket again.
- Sam Ruby
Re: agenda websocket seems to be down
Posted by Sam Ruby <ru...@intertwingly.net>.
On Tue, Oct 6, 2020 at 1:10 PM Roy T. Fielding <fi...@gbiv.com> wrote:
>
> > On Oct 6, 2020, at 4:31 AM, sebb <se...@gmail.com> wrote:
> >
> > Apart from a lack of console messages, how can one check if the socket
> > is working properly?
>
> Maybe try something that fills in a form, like sending reminders.
> You can cancel it after the form is filled, but the filling
> seems to be done via websocket. I don't know if we are logging
> the requests received via a websocket.
>
> I was able to send out the reminders on Monday, so your fix
> worked for me.
Random, disjointed thoughts, in no particular order.
1) The only use of websockets by the board agenda application is for
asynchronous, one-way messages from the server to the client,
generally to provide updates to data that the client already has a
stale version of. It would surprise me if being unable to obtain a
websocket connection would affect forms in any way other than in the
staleness of the content. (Note: I'm saying "it would surprise me",
not that it can't happen. I'm often surprised).
2) The problems to date could all have been detected by simply
attempting to open a websocket connection, nothing more is needed. If
the socket can't be opened, it clearly is a problem. Down the road,
it may be worth exploring additional tests, but this single test could
be a good place to start.
3) Prior to websockets, I tried both eventsource and websockets hosted
by the Apache webserver itself. Both lead to lock ups and even to
crashing the VM due to out-of-memory errors. When I moved the
websocket to a separate process and port, things worked better.
Eventually the infra team firewalled that port, and I had the Apache
webserver forward the requests to the process. When the client
connected to the websocket process directly it made a lot of sense for
that process to implement SSL. Now that that is no longer the case,
it no longer is necessary. In fact, it no longer is necessary for the
websocket to be running as root, as that is only done to enable it to
access the certificates.
> ....Roy
- Sam Ruby
> > On Mon, 5 Oct 2020 at 17:08, sebb <se...@gmail.com> wrote:
> >>
> >> As you suspected, the failure was:
> >> Cert does not match for name 'localhost'
> >>
> >> This fixes it, but may not be the best solution:
> >> SSLProxyCheckPeerName off
>
>
Re: agenda websocket seems to be down
Posted by "Roy T. Fielding" <fi...@gbiv.com>.
> On Oct 6, 2020, at 4:31 AM, sebb <se...@gmail.com> wrote:
>
> Apart from a lack of console messages, how can one check if the socket
> is working properly?
Maybe try something that fills in a form, like sending reminders.
You can cancel it after the form is filled, but the filling
seems to be done via websocket. I don't know if we are logging
the requests received via a websocket.
I was able to send out the reminders on Monday, so your fix
worked for me.
....Roy
> On Mon, 5 Oct 2020 at 17:08, sebb <se...@gmail.com> wrote:
>>
>> As you suspected, the failure was:
>> Cert does not match for name 'localhost'
>>
>> This fixes it, but may not be the best solution:
>> SSLProxyCheckPeerName off
Re: agenda websocket seems to be down
Posted by sebb <se...@gmail.com>.
Apart from a lack of console messages, how can one check if the socket
is working properly?
On Mon, 5 Oct 2020 at 17:08, sebb <se...@gmail.com> wrote:
>
> As you suspected, the failure was:
> Cert does not match for name 'localhost'
>
> This fixes it, but may not be the best solution:
> SSLProxyCheckPeerName off
>
> On Sat, 3 Oct 2020 at 15:20, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > This is unrelated to the problem I was seeing (the websocket server
> > was failing).
> >
> > The message in the error log:
> >
> > [Sat Oct 03 13:52:54.956298 2020] [proxy:error] [pid 1042:tid
> > 140478557177600] [client 71.204.185.91:59703] AH00898: Error during
> > SSL Handshake with remote server returned by /board/agenda/websocket/
> >
> > The relevant portion of the Apache configuration:
> >
> > SSLProxyEngine On
> > ProxyPass "/board/agenda/websocket/" "wss://localhost:34234/"
> >
> > So it looks like the Apache web server is trying to create a SSL
> > connection to a localhost websocket, and isn't happy about something,
> > most likely the certificate as it isn't possible to create a
> > certificate for localhost. But whimsy-vm4 is set up the same way, and
> > works. Not sure what the difference is, perhaps the newer libraries
> > are actually checking the certificate now, or perhaps the base puppet
> > configuration is different.
> >
> > The following seems to have some suggestions worth exploring:
> >
> > https://serverfault.com/questions/538086/proxyerror-ah00898-error-during-ssl-handshake-with-remote-server
> >
> > - Sam Ruby
> >
> > P.S. It does seem odd that the whimsy_error.log is empty and the
> > errors are going to whimsy_error.log.1.
> >
> > On Fri, Oct 2, 2020 at 2:43 PM Roy T. Fielding <fi...@gbiv.com> wrote:
> > >
> > > I was going to send out reminders under "missing" but the agenda
> > > won't fill the email form after clicking on "send initial reminders".
> > >
> > > My web console shows 500 errors every few minutes
> > >
> > > GET wss://whimsy.apache.org/board/agenda/websocket/[HTTP/1.1 500 Internal Server Error 763ms]
> > >
> > > GET
> > > wss://whimsy.apache.org/board/agenda/websocket/
> > > Status
> > > 500
> > > Internal Server Error
> > > VersionHTTP/1.1
> > > Transferred253 B (0 B size)
> > > Accept-Ranges
> > > bytes
> > > Connection
> > > close
> > > Content-Length
> > > 5220
> > > Content-Type
> > > text/html
> > > Date
> > > Fri, 02 Oct 2020 18:37:16 GMT
> > >
> > > I would guess this is the issue that Sam talked about earlier,
> > > but I thought it was fixed already.
> > >
> > > Er, looks like the error logs just rotated. Is some work being done?
> > >
> > > ....Roy
> > >
Re: agenda websocket seems to be down
Posted by sebb <se...@gmail.com>.
As you suspected, the failure was:
Cert does not match for name 'localhost'
This fixes it, but may not be the best solution:
SSLProxyCheckPeerName off
On Sat, 3 Oct 2020 at 15:20, Sam Ruby <ru...@intertwingly.net> wrote:
>
> This is unrelated to the problem I was seeing (the websocket server
> was failing).
>
> The message in the error log:
>
> [Sat Oct 03 13:52:54.956298 2020] [proxy:error] [pid 1042:tid
> 140478557177600] [client 71.204.185.91:59703] AH00898: Error during
> SSL Handshake with remote server returned by /board/agenda/websocket/
>
> The relevant portion of the Apache configuration:
>
> SSLProxyEngine On
> ProxyPass "/board/agenda/websocket/" "wss://localhost:34234/"
>
> So it looks like the Apache web server is trying to create a SSL
> connection to a localhost websocket, and isn't happy about something,
> most likely the certificate as it isn't possible to create a
> certificate for localhost. But whimsy-vm4 is set up the same way, and
> works. Not sure what the difference is, perhaps the newer libraries
> are actually checking the certificate now, or perhaps the base puppet
> configuration is different.
>
> The following seems to have some suggestions worth exploring:
>
> https://serverfault.com/questions/538086/proxyerror-ah00898-error-during-ssl-handshake-with-remote-server
>
> - Sam Ruby
>
> P.S. It does seem odd that the whimsy_error.log is empty and the
> errors are going to whimsy_error.log.1.
>
> On Fri, Oct 2, 2020 at 2:43 PM Roy T. Fielding <fi...@gbiv.com> wrote:
> >
> > I was going to send out reminders under "missing" but the agenda
> > won't fill the email form after clicking on "send initial reminders".
> >
> > My web console shows 500 errors every few minutes
> >
> > GET wss://whimsy.apache.org/board/agenda/websocket/[HTTP/1.1 500 Internal Server Error 763ms]
> >
> > GET
> > wss://whimsy.apache.org/board/agenda/websocket/
> > Status
> > 500
> > Internal Server Error
> > VersionHTTP/1.1
> > Transferred253 B (0 B size)
> > Accept-Ranges
> > bytes
> > Connection
> > close
> > Content-Length
> > 5220
> > Content-Type
> > text/html
> > Date
> > Fri, 02 Oct 2020 18:37:16 GMT
> >
> > I would guess this is the issue that Sam talked about earlier,
> > but I thought it was fixed already.
> >
> > Er, looks like the error logs just rotated. Is some work being done?
> >
> > ....Roy
> >
Re: agenda websocket seems to be down
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Tue, Oct 20, 2020 at 2:43 AM Roy T. Fielding <fi...@gbiv.com> wrote:
> ...I don't have this problem on Safari. It works fine.
> In fact, it doesn't even seem to be using the websocket connection...
FWIW, Safari worked for me indeed right now, after Firefox (macosx)
was blocking when trying to commit a comment.
-Bertrand
Re: agenda websocket seems to be down
Posted by "Roy T. Fielding" <fi...@gbiv.com>.
> On Oct 3, 2020, at 7:20 AM, Sam Ruby <ru...@intertwingly.net> wrote:
>
> This is unrelated to the problem I was seeing (the websocket server
> was failing).
>
> The message in the error log:
>
> [Sat Oct 03 13:52:54.956298 2020] [proxy:error] [pid 1042:tid
> 140478557177600] [client 71.204.185.91:59703] AH00898: Error during
> SSL Handshake with remote server returned by /board/agenda/websocket/
>
> The relevant portion of the Apache configuration:
>
> SSLProxyEngine On
> ProxyPass "/board/agenda/websocket/" "wss://localhost:34234/"
>
> So it looks like the Apache web server is trying to create a SSL
> connection to a localhost websocket, and isn't happy about something,
> most likely the certificate as it isn't possible to create a
> certificate for localhost. But whimsy-vm4 is set up the same way, and
> works. Not sure what the difference is, perhaps the newer libraries
> are actually checking the certificate now, or perhaps the base puppet
> configuration is different.
>
> The following seems to have some suggestions worth exploring:
>
> https://serverfault.com/questions/538086/proxyerror-ah00898-error-during-ssl-handshake-with-remote-server
>
> - Sam Ruby
>
> P.S. It does seem odd that the whimsy_error.log is empty and the
> errors are going to whimsy_error.log.1.
So, I've been overwhelmed with non-ASF stuff and haven't figured
this out, but I have been working around it in a couple ways.
First, it only seems to happen after the first request times out.
As in, I can quit Firefox, reopen, go to the agenda, and perform
one sequence of requests successfully if I am fast enough.
That's how I managed to send out the final reminders.
But then the websocket times out and I cannot do anything --
a page reload will simply block, even though it isn't going
to reload the page via websocket (AFAIK).
Second, I don't have this problem on Safari. It works fine.
In fact, it doesn't even seem to be using the websocket connection.
I haven't tried other browsers yet.
....Roy
Re: agenda websocket seems to be down
Posted by Sam Ruby <ru...@intertwingly.net>.
This is unrelated to the problem I was seeing (the websocket server
was failing).
The message in the error log:
[Sat Oct 03 13:52:54.956298 2020] [proxy:error] [pid 1042:tid
140478557177600] [client 71.204.185.91:59703] AH00898: Error during
SSL Handshake with remote server returned by /board/agenda/websocket/
The relevant portion of the Apache configuration:
SSLProxyEngine On
ProxyPass "/board/agenda/websocket/" "wss://localhost:34234/"
So it looks like the Apache web server is trying to create a SSL
connection to a localhost websocket, and isn't happy about something,
most likely the certificate as it isn't possible to create a
certificate for localhost. But whimsy-vm4 is set up the same way, and
works. Not sure what the difference is, perhaps the newer libraries
are actually checking the certificate now, or perhaps the base puppet
configuration is different.
The following seems to have some suggestions worth exploring:
https://serverfault.com/questions/538086/proxyerror-ah00898-error-during-ssl-handshake-with-remote-server
- Sam Ruby
P.S. It does seem odd that the whimsy_error.log is empty and the
errors are going to whimsy_error.log.1.
On Fri, Oct 2, 2020 at 2:43 PM Roy T. Fielding <fi...@gbiv.com> wrote:
>
> I was going to send out reminders under "missing" but the agenda
> won't fill the email form after clicking on "send initial reminders".
>
> My web console shows 500 errors every few minutes
>
> GET wss://whimsy.apache.org/board/agenda/websocket/[HTTP/1.1 500 Internal Server Error 763ms]
>
> GET
> wss://whimsy.apache.org/board/agenda/websocket/
> Status
> 500
> Internal Server Error
> VersionHTTP/1.1
> Transferred253 B (0 B size)
> Accept-Ranges
> bytes
> Connection
> close
> Content-Length
> 5220
> Content-Type
> text/html
> Date
> Fri, 02 Oct 2020 18:37:16 GMT
>
> I would guess this is the issue that Sam talked about earlier,
> but I thought it was fixed already.
>
> Er, looks like the error logs just rotated. Is some work being done?
>
> ....Roy
>