You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by T Smith <ai...@gmail.com> on 2018/06/18 17:49:51 UTC
Re: socket.io in knox
Hi Sandeep,
Back to fighting with this. Through some nginx debugging on my backend I've
come to the conclusion that Knox isn't sending the query parameters on to
the backend, regardless of what rewrite rules I specify.
I.e. I send /gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=
polling&t=MDGlYhd
<http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
Knox sends /socket.io/
<http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
This yields a 400 error, and Knox drops the websocket to the backend. What
was confusing me was the 101 in the browser, but I see now that Knox is
terminating the connection with the browser on one side and opening another
connection with the backend on the other.
I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc but
I'm wondering if this code needs further work to support query parameters?
What do you think?
Thanks in advance!
/ailuropod4
On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com> wrote:
> Hi Sandeep,
>
> Looking at my nginx server, it never sees the transport=websocket ws://
> protocol request, but it does see a http request for / at the corresponding
> time, so I think perhaps the root problem here is in that rewrite, as you
> mention. By the way, the reason for ws:// and not wss:// is I switched off
> TLS to remove another potential source of issue, so it's all plain http. I
> saw the same behaviour over https.
>
> My services, now, look like this -
>
> <route path="/pndaconsole/socket.io">
> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
> to="request.url"/>
> </route>
>
> <route path="/pndaconsole/metrics">
> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
> to="request.url"/>
> </route>
>
> <route path="/pndaconsole/">
> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
> to="request.url"/>
> </route>
>
> <route path="/pndaconsole/**">
> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
> to="request.url"/>
> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
> to="response.body"/>
> </route>
>
> My inbound rewrites, now, look like this -
>
> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
> pattern="*://*:*/**/pndaconsole/">
> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
> </rule>
>
> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
> <http://socket.io/?%7B**%7D>">
> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
> <http://socket.io/?%7B**%7D>"/>
> </rule>
>
> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
> pattern="*://*:*/**/pndaconsole/{**}">
> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
> </rule>
>
> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/>
> </rule>
>
> The ws://...transport=websocket requests are rewritten as /, so I tried
> adding an entry in my topology for WEBSOCKET and an additional inbound rule
> like this -
>
> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
> <http://socket.io/?%7B**%7D>">
> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
> <http://socket.io/?%7B**%7D>"/>
> </rule>
>
> Where the topology entry is the same as PNDACONSOLE except it uses the
> ws:// protocol specifier. This didn't work at all, and in fact at that
> point, nothing got through and everything fell through to the default
> rewrite rule.
>
> I've attached the gateway log with debug turned on everywhere (tarballed
> as it's huge).
>
> I guess what I'm missing is a clear idea of what I'm supposed to do in the
> service/rewrite rules to deal with the websockets request which will arrive
> with a different protocol specifier and a different query parameter. Should
> I explicitly map those through to the back end, or is this something Knox
> handles internally?
>
> Any insight into this appreciated, it seems very close to working, and in
> fact socket.io does it's usual fallback so it is functional after a
> fashion but it seems like this should fully work given what I see with
> Zeppelin.
>
>
> Cheers,
> /ailuropod4
>
>
> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <mo...@gmail.com>
> wrote:
>
>> Hello ailuropod4
>>
>> You should not have to reroute to a different host port, this should
>> work.
>>
>> Looking at the responses, it looks like protocol switching happens at the
>> Knox end, between browser and Knox (going by the url ws://
>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?
>> EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW) but it should have
>> been wss:// wonder why this is ws://.
>>
>> Also, can you include seperate "ws" rules for your service, similar to
>> Zeppelin.
>> Also, make sure you have, inbound routes in service.xml for e.g.
>>
>> <routes>
>> <route path="/zeppelin/ws">
>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound" to="request.url"/>
>> </route>
>>
>> <route path="/zeppelin/ws**">
>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/>
>> </route>
>> </routes>
>>
>> and then reference it in rewrite.xml.
>>
>> Looking at the error, it looks like the issue is not with rewrite rules
>> but with the connection between Knox and the backend.
>> We can see that Knox is initiating an upgrade request but then for some
>> reason the backend service is closing/refusing it.
>> Can you turn up the logging in Knox to Debug and share it if you can
>> (this should atleast tell us whether rewrite rules are working).
>>
>> Also, do you see any errors at the backend service ? it woud be good to
>> know what the backend is seeing and reasons for it to refuse the upgrade
>> and also the error code
>> Unfortunately the error code is gobbled up somewhere in the process.
>>
>> Best,
>> Sandeep
>>
>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com> wrote:
>>
>>> And if it helps, the stack trace as a result of this from Knox follows.
>>>
>>> Can I match a url containing &transport=websocket in one of the query
>>> parameters and map it through to a service defined as ws:// in the
>>> topology? At the moment, as above, everything is sent to a http://
>>> service, maybe this is the crux of the problem?
>>>
>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch
>>> protocols
>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to
>>> connect to websocket server: java.io.IOException: Connect failure
>>> java.io.IOException: Connect failure
>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(C
>>> lientContainer.java:157)
>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connectTo
>>> Server(ClientContainer.java:180)
>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW
>>> ebSocketConnect(ProxyWebSocketAdapter.java:97)
>>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent
>>> Driver.onConnect(JettyListenerEventDriver.java:87)
>>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive
>>> r.openSession(AbstractEventDriver.java:227)
>>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(Web
>>> SocketSession.java:421)
>>> at org.eclipse.jetty.websocket.server.WebSocketServerConnection
>>> .onOpen(WebSocketServerConnection.java:72)
>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoi
>>> nt.java:185)
>>> at org.eclipse.jetty.server.HttpConnection.completed(HttpConnec
>>> tion.java:345)
>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java
>>> :436)
>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>>> ction.java:257)
>>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec
>>> tion.java:544)
>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>>> ThreadPool.java:635)
>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT
>>> hreadPool.java:555)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>> switch protocols
>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.vali
>>> dateResponse(UpgradeConnection.java:314)
>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.read
>>> (UpgradeConnection.java:241)
>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.onFi
>>> llable(UpgradeConnection.java:163)
>>> ... 4 more
>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing
>>> connection)
>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException: Connect
>>> failure
>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW
>>> ebSocketConnect(ProxyWebSocketAdapter.java:106)
>>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent
>>> Driver.onConnect(JettyListenerEventDriver.java:87)
>>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive
>>> r.openSession(AbstractEventDriver.java:227)
>>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(Web
>>> SocketSession.java:421)
>>> at org.eclipse.jetty.websocket.server.WebSocketServerConnection
>>> .onOpen(WebSocketServerConnection.java:72)
>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoi
>>> nt.java:185)
>>> at org.eclipse.jetty.server.HttpConnection.completed(HttpConnec
>>> tion.java:345)
>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java
>>> :436)
>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>>> ction.java:257)
>>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec
>>> tion.java:544)
>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>>> ThreadPool.java:635)
>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT
>>> hreadPool.java:555)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: java.io.IOException: Connect failure
>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(C
>>> lientContainer.java:157)
>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connectTo
>>> Server(ClientContainer.java:180)
>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW
>>> ebSocketConnect(ProxyWebSocketAdapter.java:97)
>>> ... 12 more
>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>> switch protocols
>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.vali
>>> dateResponse(UpgradeConnection.java:314)
>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.read
>>> (UpgradeConnection.java:241)
>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.onFi
>>> llable(UpgradeConnection.java:163)
>>> ... 4 more
>>>
>>>
>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com> wrote:
>>>
>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>
>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com> wrote:
>>>>
>>>>> Hi Sandeep,
>>>>>
>>>>> Here's what's happening -
>>>>>
>>>>>
>>>>> 1. Request URL:
>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io
>>>>> /?EIO=3&transport=polling&t=MDGlYhd
>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>> 2. Request Method:
>>>>> GET
>>>>> 3. Status Code:
>>>>> 200 OK
>>>>>
>>>>>
>>>>> 1. Request URL:
>>>>> ws://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?
>>>>> EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW>
>>>>> 2. Request Method:
>>>>> GET
>>>>> 3. Status Code:
>>>>> 101 Switching Protocols
>>>>>
>>>>> So far, so good. But, the next couple of requests fail, with 400 or
>>>>> 500 range errors, and the gateway log complains about protocol errors. I
>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>> big picture across.
>>>>>
>>>>> Looking at tcpdump, what's happening is that ws:// request is being
>>>>> nerfed to a GET /, which on my backend happens to resolve to a nice front
>>>>> page of HTML. Websockets obviously panics as this isn't what it expected,
>>>>> and then the whole process repeats. On my backend, /socket.io is
>>>>> routed to the socket endpoint, whereas / isn't. So, my theory is that if I
>>>>> can somehow convince Knox to send out the ws:// request with the correct
>>>>> path including /socket.io, it might work.
>>>>>
>>>>> At the moment my service/rewrite sections are very simple, I pick up
>>>>> requests with socket.io and send them off to the backend.
>>>>>
>>>>> services.xml -
>>>>>
>>>>> <policies>
>>>>> <policy role="webappsec"/>
>>>>> <policy role="authentication" name="Anonymous"/>
>>>>> <policy role="rewrite"/>
>>>>> <policy role="authorization"/>
>>>>> </policies>
>>>>> <routes>
>>>>> <route path="/pndaconsole">
>>>>> </route>
>>>>> <route path="/pndaconsole/**">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>> to="response.body"/>
>>>>> </route>
>>>>> </routes>
>>>>>
>>>>> rewrite.xml -
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>> </rule>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>> </rule>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>> </rule>
>>>>> (I've snipped the rest of the rules handling outbound rewrites as I
>>>>> don't believe they're relevant, but let me know if you think otherwise)
>>>>>
>>>>> By all accounts it looks like it's *nearly* working, but I'm stumped
>>>>> on where my /socket.io path is going. I was considering routing all
>>>>> websockets stuff to a completely separate backend/port so that it would go
>>>>> to the right place regardless of how it was rewritten, but that seems a bit
>>>>> extreme and I was hoping to use this approach.
>>>>>
>>>>> My end to end is -
>>>>>
>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>> websockets backend in nodejs
>>>>>
>>>>> --> flat files, being an angularjs app
>>>>>
>>>>> We don't have to use socket.io, I guess, but it is currently used
>>>>> everywhere so I'd rather avoid swapping that out to get this working if
>>>>> possible. It's a fairly common library.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am not very familiar with socket.io apps, so you might have to
>>>>>> fill me in about what you are trying to do.
>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>
>>>>>> What version of Knox are you using ?
>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has some
>>>>>> examples in the comments.
>>>>>>
>>>>>> What is the ws:// url you are trying to rewrite and the rules you are
>>>>>> using ?
>>>>>>
>>>>>> Best,
>>>>>> Sandeep
>>>>>>
>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'm struggling to get a simple socket.io based application working
>>>>>>> through Knox.
>>>>>>>
>>>>>>> I see websockets is supported, but the upgrade handshake seems to be
>>>>>>> nerfed on the way out, specifically the /socket.io/? etc part
>>>>>>> simply becomes /.
>>>>>>>
>>>>>>> I've tried lots of permutations - does anyone have a working example
>>>>>>> they can point me towards? I've looked at the Zeppelin approach, but this
>>>>>>> isn't quite the same thing (not socket.io).
>>>>>>>
>>>>>>> Any help appreciated!
>>>>>>>
>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: socket.io in knox
Posted by T Smith <ai...@gmail.com>.
Hi Sandeep,
It's going to take me a day or two to clear the time to get back around to
this, in the meantime I've raised
https://issues.apache.org/jira/browse/KNOX-1361 (under my more usual
github/jira handle).
Thanks!
On Wed, Jun 20, 2018 at 2:47 PM, Sandeep Moré <mo...@gmail.com> wrote:
> I was thinking about the issue (c) I think the issue here is with the way
> GatewayWebsocketHandler is added in GatewayServer.createhandler() class. I
> am thinking the reason why the rewrite rules are skipped is because
> GatewayWebsocketHandler is called before other handlers, we need to test
> the order in which this handler is added.
>
> Best,
> Sandeep
>
> On Wed, Jun 20, 2018 at 1:13 AM Sandeep Moré <mo...@gmail.com>
> wrote:
>
>> Hello ailuropod4,
>>
>> Excellent analysis !
>> you are right about (a), (b), (c) and (d).
>> (a) and (b) can be easily fixed, we should have caught this earlier :( we
>> do need to account for url query string and url fragment, I am thinking, in
>> the createWebSocket() method we can extract query and fragment along with
>> path and concatenate it to path, we then use this path to pass it down to
>> getMatchedBackendURL(), like you pointed out this should hopefully help
>> you.
>>
>> It would be awesome if you could create a JIRA for this issue (if we do
>> not have already) and create a patch along with a UnitTest, this would be
>> extremely useful and should take care of (a) and (b)
>>
>> I think we need to look closely at (c), can you open a JIRA for (c) as
>> well so we can investigate. I was hoping for the request to hit
>> UrlRewriteServletFilter.doFilter() before reaching websocket handler,
>> specifically hitting the following call
>>
>> UrlRewriteRequest rewriteRequest = new UrlRewriteRequest( config, request
>> );
>>
>> From your observation it appears that this might not be the case which is
>> a bug !
>> I might not be able to look at it closely for few more days, it would be
>> awesome if you could also take a look at it.
>>
>> Again thanks for pointing these issues :)
>>
>> Best,
>> Sandeep
>>
>> On Tue, Jun 19, 2018 at 4:52 AM T Smith <ai...@gmail.com> wrote:
>>
>>> Hi Sandeep,
>>>
>>> I've had a look at this and here's what I think - maybe we should take
>>> this to knox-developers, I'm happy to help work out a solution but I'm
>>> going to need some insight into the design intent around this module.
>>>
>>> a) The existing unit test is hard-coded to test a backend URL that ends
>>> in /ws. This corresponds to a special case in the code for Zeppelin, so the
>>> other paths are never tested, and hence the unit test always passes.
>>> b) createWebSocket passes only a path, so the query component could
>>> never be considered by getMatchedBackendURL
>>> c) getMatchedBackendURL doesn't seem to base its logic on the rewrite
>>> rules at all, in the case where the backend URL doesn't end in /ws it
>>> appends the remainder of the path to the backend (you can demonstrate this
>>> by altering the test case to remove the /ws from the backend URL and
>>> running the existing test - the rewrite rule is {**}/channels but the
>>> result is simply /channels).
>>> d) Due to (a) this is a moot point, but the unit test doesn't check that
>>> the path was rewritten as expected, so it will pass regardless.
>>>
>>> I think if we address (b) to pass a concatenation of the path and query
>>> then it might work for cases like mine, but this doesn't address (c) which
>>> will affect anyone wanting to control rewrites. I guess addressing (a) and
>>> (d) would help in the longer term.
>>>
>>> Hope this makes sense and I haven't completely missed the point :-) It
>>> does explain the behaviour I'm seeing. How would you like to proceed here?
>>>
>>> Cheers,
>>> /ailuropod4
>>>
>>>
>>>
>>>
>>> On Mon, Jun 18, 2018 at 7:26 PM, Sandeep Moré <mo...@gmail.com>
>>> wrote:
>>>
>>>> may be you are right about not not properly handeling the query params
>>>> and we might need to fix this regex:
>>>>
>>>> final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
>>>>
>>>> You should be able to do a quick test using a this Unit Test
>>>> <https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jun 18, 2018 at 1:55 PM T Smith <ai...@gmail.com> wrote:
>>>>
>>>>> Sorry, bad example.
>>>>>
>>>>> I send ws://<host>:<port>/gateway/pnda/pndaconsole/socke
>>>>> t.io/?EIO=3&transport=websocket
>>>>>
>>>>> Knox sends /socket.io/
>>>>>
>>>>> That yields a 400 error, Knox throws org.eclipse.jetty.websocket.api.UpgradeException:
>>>>> Didn't switch protocols
>>>>>
>>>>>
>>>>> On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ai...@gmail.com> wrote:
>>>>>
>>>>>> Hi Sandeep,
>>>>>>
>>>>>> Back to fighting with this. Through some nginx debugging on my
>>>>>> backend I've come to the conclusion that Knox isn't sending the query
>>>>>> parameters on to the backend, regardless of what rewrite rules I specify.
>>>>>>
>>>>>> I.e. I send /gateway/pnda/pndaconsole/socket.io/?EIO=3&
>>>>>> transport=polling&t=MDGlYhd
>>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>>
>>>>>> Knox sends /socket.io/
>>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>>
>>>>>> This yields a 400 error, and Knox drops the websocket to the backend.
>>>>>> What was confusing me was the 101 in the browser, but I see now that Knox
>>>>>> is terminating the connection with the browser on one side and opening
>>>>>> another connection with the backend on the other.
>>>>>>
>>>>>> I see this https://git-wip-us.apache.org/repos/asf?p=knox.
>>>>>> git;h=98a08fc but I'm wondering if this code needs further work to
>>>>>> support query parameters? What do you think?
>>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> /ailuropod4
>>>>>>
>>>>>> On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Sandeep,
>>>>>>>
>>>>>>> Looking at my nginx server, it never sees the transport=websocket
>>>>>>> ws:// protocol request, but it does see a http request for / at the
>>>>>>> corresponding time, so I think perhaps the root problem here is in that
>>>>>>> rewrite, as you mention. By the way, the reason for ws:// and not wss:// is
>>>>>>> I switched off TLS to remove another potential source of issue, so it's all
>>>>>>> plain http. I saw the same behaviour over https.
>>>>>>>
>>>>>>> My services, now, look like this -
>>>>>>>
>>>>>>> <route path="/pndaconsole/socket.io">
>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>> to="request.url"/>
>>>>>>> </route>
>>>>>>>
>>>>>>> <route path="/pndaconsole/metrics">
>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>>>> to="request.url"/>
>>>>>>> </route>
>>>>>>>
>>>>>>> <route path="/pndaconsole/">
>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>> to="request.url"/>
>>>>>>> </route>
>>>>>>>
>>>>>>> <route path="/pndaconsole/**">
>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>> to="request.url"/>
>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>>> to="response.body"/>
>>>>>>> </route>
>>>>>>>
>>>>>>> My inbound rewrites, now, look like this -
>>>>>>>
>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>>> </rule>
>>>>>>>
>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/
>>>>>>> socket.io/?{**} <http://socket.io/?%7B**%7D>"/>
>>>>>>> </rule>
>>>>>>>
>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>>> </rule>
>>>>>>>
>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>>>> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>>>>>>> <rewrite template="{$serviceUrl[
>>>>>>> PNDACONSOLE]}/metrics/?{**}"/>
>>>>>>> </rule>
>>>>>>>
>>>>>>> The ws://...transport=websocket requests are rewritten as /, so I
>>>>>>> tried adding an entry in my topology for WEBSOCKET and an additional
>>>>>>> inbound rule like this -
>>>>>>>
>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
>>>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>>>> </rule>
>>>>>>>
>>>>>>> Where the topology entry is the same as PNDACONSOLE except it uses
>>>>>>> the ws:// protocol specifier. This didn't work at all, and in fact at that
>>>>>>> point, nothing got through and everything fell through to the default
>>>>>>> rewrite rule.
>>>>>>>
>>>>>>> I've attached the gateway log with debug turned on everywhere
>>>>>>> (tarballed as it's huge).
>>>>>>>
>>>>>>> I guess what I'm missing is a clear idea of what I'm supposed to do
>>>>>>> in the service/rewrite rules to deal with the websockets request which will
>>>>>>> arrive with a different protocol specifier and a different query parameter.
>>>>>>> Should I explicitly map those through to the back end, or is this something
>>>>>>> Knox handles internally?
>>>>>>>
>>>>>>> Any insight into this appreciated, it seems very close to working,
>>>>>>> and in fact socket.io does it's usual fallback so it is functional
>>>>>>> after a fashion but it seems like this should fully work given what I see
>>>>>>> with Zeppelin.
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> /ailuropod4
>>>>>>>
>>>>>>>
>>>>>>> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <moresandeep@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hello ailuropod4
>>>>>>>>
>>>>>>>> You should not have to reroute to a different host port, this
>>>>>>>> should work.
>>>>>>>>
>>>>>>>> Looking at the responses, it looks like protocol switching happens
>>>>>>>> at the Knox end, between browser and Knox (going by the url ws://
>>>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/
>>>>>>>> socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW) but
>>>>>>>> it should have
>>>>>>>> been wss:// wonder why this is ws://.
>>>>>>>>
>>>>>>>> Also, can you include seperate "ws" rules for your service, similar
>>>>>>>> to Zeppelin.
>>>>>>>> Also, make sure you have, inbound routes in service.xml for e.g.
>>>>>>>>
>>>>>>>> <routes>
>>>>>>>> <route path="/zeppelin/ws">
>>>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound"
>>>>>>>> to="request.url"/>
>>>>>>>> </route>
>>>>>>>>
>>>>>>>> <route path="/zeppelin/ws**">
>>>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/inbound"
>>>>>>>> to="request.url"/>
>>>>>>>> </route>
>>>>>>>> </routes>
>>>>>>>>
>>>>>>>> and then reference it in rewrite.xml.
>>>>>>>>
>>>>>>>> Looking at the error, it looks like the issue is not with rewrite
>>>>>>>> rules but with the connection between Knox and the backend.
>>>>>>>> We can see that Knox is initiating an upgrade request but then for
>>>>>>>> some reason the backend service is closing/refusing it.
>>>>>>>> Can you turn up the logging in Knox to Debug and share it if you
>>>>>>>> can (this should atleast tell us whether rewrite rules are working).
>>>>>>>>
>>>>>>>> Also, do you see any errors at the backend service ? it woud be
>>>>>>>> good to know what the backend is seeing and reasons for it to refuse the
>>>>>>>> upgrade and also the error code
>>>>>>>> Unfortunately the error code is gobbled up somewhere in the process.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Sandeep
>>>>>>>>
>>>>>>>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> And if it helps, the stack trace as a result of this from Knox
>>>>>>>>> follows.
>>>>>>>>>
>>>>>>>>> Can I match a url containing &transport=websocket in one of the
>>>>>>>>> query parameters and map it through to a service defined as ws:// in the
>>>>>>>>> topology? At the moment, as above, everything is sent to a http://
>>>>>>>>> service, maybe this is the crux of the problem?
>>>>>>>>>
>>>>>>>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>>>>>>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>>>>>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch
>>>>>>>>> protocols
>>>>>>>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>>>>>>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to
>>>>>>>>> connect to websocket server: java.io.IOException: Connect failure
>>>>>>>>> java.io.IOException: Connect failure
>>>>>>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.
>>>>>>>>> connect(ClientContainer.java:157)
>>>>>>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.
>>>>>>>>> connectToServer(ClientContainer.java:180)
>>>>>>>>> at org.apache.knox.gateway.websockets.
>>>>>>>>> ProxyWebSocketAdapter.onWebSocketConnect(
>>>>>>>>> ProxyWebSocketAdapter.java:97)
>>>>>>>>> at org.eclipse.jetty.websocket.common.events.
>>>>>>>>> JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:
>>>>>>>>> 87)
>>>>>>>>> at org.eclipse.jetty.websocket.common.events.
>>>>>>>>> AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>>>> at org.eclipse.jetty.websocket.server.
>>>>>>>>> WebSocketServerConnection.onOpen(WebSocketServerConnection.
>>>>>>>>> java:72)
>>>>>>>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(
>>>>>>>>> AbstractEndPoint.java:185)
>>>>>>>>> at org.eclipse.jetty.server.HttpConnection.completed(
>>>>>>>>> HttpConnection.java:345)
>>>>>>>>> at org.eclipse.jetty.server.HttpChannel.handle(
>>>>>>>>> HttpChannel.java:436)
>>>>>>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(
>>>>>>>>> HttpConnection.java:257)
>>>>>>>>> at org.eclipse.jetty.io.AbstractConnection$2.run(
>>>>>>>>> AbstractConnection.java:544)
>>>>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>>>>>>>> QueuedThreadPool.java:635)
>>>>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>>>>>>>>> QueuedThreadPool.java:555)
>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException:
>>>>>>>>> Didn't switch protocols
>>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>>> ient.io.UpgradeConnection.validateResponse(
>>>>>>>>> UpgradeConnection.java:314)
>>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>>>> ... 4 more
>>>>>>>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>>>>>>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error
>>>>>>>>> (closing connection)
>>>>>>>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException:
>>>>>>>>> Connect failure
>>>>>>>>> at org.apache.knox.gateway.websockets.
>>>>>>>>> ProxyWebSocketAdapter.onWebSocketConnect(
>>>>>>>>> ProxyWebSocketAdapter.java:106)
>>>>>>>>> at org.eclipse.jetty.websocket.common.events.
>>>>>>>>> JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:
>>>>>>>>> 87)
>>>>>>>>> at org.eclipse.jetty.websocket.common.events.
>>>>>>>>> AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>>>> at org.eclipse.jetty.websocket.server.
>>>>>>>>> WebSocketServerConnection.onOpen(WebSocketServerConnection.
>>>>>>>>> java:72)
>>>>>>>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(
>>>>>>>>> AbstractEndPoint.java:185)
>>>>>>>>> at org.eclipse.jetty.server.HttpConnection.completed(
>>>>>>>>> HttpConnection.java:345)
>>>>>>>>> at org.eclipse.jetty.server.HttpChannel.handle(
>>>>>>>>> HttpChannel.java:436)
>>>>>>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(
>>>>>>>>> HttpConnection.java:257)
>>>>>>>>> at org.eclipse.jetty.io.AbstractConnection$2.run(
>>>>>>>>> AbstractConnection.java:544)
>>>>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>>>>>>>> QueuedThreadPool.java:635)
>>>>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>>>>>>>>> QueuedThreadPool.java:555)
>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>> Caused by: java.io.IOException: Connect failure
>>>>>>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.
>>>>>>>>> connect(ClientContainer.java:157)
>>>>>>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.
>>>>>>>>> connectToServer(ClientContainer.java:180)
>>>>>>>>> at org.apache.knox.gateway.websockets.
>>>>>>>>> ProxyWebSocketAdapter.onWebSocketConnect(
>>>>>>>>> ProxyWebSocketAdapter.java:97)
>>>>>>>>> ... 12 more
>>>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException:
>>>>>>>>> Didn't switch protocols
>>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>>> ient.io.UpgradeConnection.validateResponse(
>>>>>>>>> UpgradeConnection.java:314)
>>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>>>> ... 4 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>>>>>>>
>>>>>>>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Sandeep,
>>>>>>>>>>>
>>>>>>>>>>> Here's what's happening -
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1. Request URL:
>>>>>>>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/
>>>>>>>>>>> socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>>>>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>>>>>>> 2. Request Method:
>>>>>>>>>>> GET
>>>>>>>>>>> 3. Status Code:
>>>>>>>>>>> 200 OK
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1. Request URL:
>>>>>>>>>>> ws://34.244.121.78:8443/gateway/pnda/pndaconsole/
>>>>>>>>>>> socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>>>>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW>
>>>>>>>>>>> 2. Request Method:
>>>>>>>>>>> GET
>>>>>>>>>>> 3. Status Code:
>>>>>>>>>>> 101 Switching Protocols
>>>>>>>>>>>
>>>>>>>>>>> So far, so good. But, the next couple of requests fail, with 400
>>>>>>>>>>> or 500 range errors, and the gateway log complains about protocol errors. I
>>>>>>>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>>>>>>>> big picture across.
>>>>>>>>>>>
>>>>>>>>>>> Looking at tcpdump, what's happening is that ws:// request is
>>>>>>>>>>> being nerfed to a GET /, which on my backend happens to resolve to a nice
>>>>>>>>>>> front page of HTML. Websockets obviously panics as this isn't what it
>>>>>>>>>>> expected, and then the whole process repeats. On my backend, /
>>>>>>>>>>> socket.io is routed to the socket endpoint, whereas / isn't.
>>>>>>>>>>> So, my theory is that if I can somehow convince Knox to send out the ws://
>>>>>>>>>>> request with the correct path including /socket.io, it might
>>>>>>>>>>> work.
>>>>>>>>>>>
>>>>>>>>>>> At the moment my service/rewrite sections are very simple, I
>>>>>>>>>>> pick up requests with socket.io and send them off to the
>>>>>>>>>>> backend.
>>>>>>>>>>>
>>>>>>>>>>> services.xml -
>>>>>>>>>>>
>>>>>>>>>>> <policies>
>>>>>>>>>>> <policy role="webappsec"/>
>>>>>>>>>>> <policy role="authentication" name="Anonymous"/>
>>>>>>>>>>> <policy role="rewrite"/>
>>>>>>>>>>> <policy role="authorization"/>
>>>>>>>>>>> </policies>
>>>>>>>>>>> <routes>
>>>>>>>>>>> <route path="/pndaconsole">
>>>>>>>>>>> </route>
>>>>>>>>>>> <route path="/pndaconsole/**">
>>>>>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>>>>>>> to="response.body"/>
>>>>>>>>>>> </route>
>>>>>>>>>>> </routes>
>>>>>>>>>>>
>>>>>>>>>>> rewrite.xml -
>>>>>>>>>>>
>>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>>>>>>> </rule>
>>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/
>>>>>>>>>>> socket.io/?{**} <http://socket.io/?%7B**%7D>"/>
>>>>>>>>>>> </rule>
>>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>>>>>>> </rule>
>>>>>>>>>>> (I've snipped the rest of the rules handling outbound rewrites
>>>>>>>>>>> as I don't believe they're relevant, but let me know if you think otherwise)
>>>>>>>>>>>
>>>>>>>>>>> By all accounts it looks like it's *nearly* working, but I'm
>>>>>>>>>>> stumped on where my /socket.io path is going. I was considering
>>>>>>>>>>> routing all websockets stuff to a completely separate backend/port so that
>>>>>>>>>>> it would go to the right place regardless of how it was rewritten, but that
>>>>>>>>>>> seems a bit extreme and I was hoping to use this approach.
>>>>>>>>>>>
>>>>>>>>>>> My end to end is -
>>>>>>>>>>>
>>>>>>>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>>>>>>>> websockets backend in nodejs
>>>>>>>>>>>
>>>>>>>>>>> --> flat files, being an angularjs app
>>>>>>>>>>>
>>>>>>>>>>> We don't have to use socket.io, I guess, but it is currently
>>>>>>>>>>> used everywhere so I'd rather avoid swapping that out to get this working
>>>>>>>>>>> if possible. It's a fairly common library.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <
>>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I am not very familiar with socket.io apps, so you might have
>>>>>>>>>>>> to fill me in about what you are trying to do.
>>>>>>>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>>>>>>>
>>>>>>>>>>>> What version of Knox are you using ?
>>>>>>>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has
>>>>>>>>>>>> some examples in the comments.
>>>>>>>>>>>>
>>>>>>>>>>>> What is the ws:// url you are trying to rewrite and the rules
>>>>>>>>>>>> you are using ?
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm struggling to get a simple socket.io based application
>>>>>>>>>>>>> working through Knox.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see websockets is supported, but the upgrade handshake seems
>>>>>>>>>>>>> to be nerfed on the way out, specifically the /socket.io/?
>>>>>>>>>>>>> etc part simply becomes /.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've tried lots of permutations - does anyone have a working
>>>>>>>>>>>>> example they can point me towards? I've looked at the Zeppelin approach,
>>>>>>>>>>>>> but this isn't quite the same thing (not socket.io).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any help appreciated!
>>>>>>>>>>>>>
>>>>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
Re: socket.io in knox
Posted by Sandeep Moré <mo...@gmail.com>.
I was thinking about the issue (c) I think the issue here is with the way
GatewayWebsocketHandler is added in GatewayServer.createhandler() class. I
am thinking the reason why the rewrite rules are skipped is because
GatewayWebsocketHandler
is called before other handlers, we need to test the order in which this
handler is added.
Best,
Sandeep
On Wed, Jun 20, 2018 at 1:13 AM Sandeep Moré <mo...@gmail.com> wrote:
> Hello ailuropod4,
>
> Excellent analysis !
> you are right about (a), (b), (c) and (d).
> (a) and (b) can be easily fixed, we should have caught this earlier :( we
> do need to account for url query string and url fragment, I am thinking, in
> the createWebSocket() method we can extract query and fragment along with
> path and concatenate it to path, we then use this path to pass it down to
> getMatchedBackendURL(), like you pointed out this should hopefully help
> you.
>
> It would be awesome if you could create a JIRA for this issue (if we do
> not have already) and create a patch along with a UnitTest, this would be
> extremely useful and should take care of (a) and (b)
>
> I think we need to look closely at (c), can you open a JIRA for (c) as
> well so we can investigate. I was hoping for the request to hit
> UrlRewriteServletFilter.doFilter() before reaching websocket handler,
> specifically hitting the following call
>
> UrlRewriteRequest rewriteRequest = new UrlRewriteRequest( config, request
> );
>
> From your observation it appears that this might not be the case which is
> a bug !
> I might not be able to look at it closely for few more days, it would be
> awesome if you could also take a look at it.
>
> Again thanks for pointing these issues :)
>
> Best,
> Sandeep
>
> On Tue, Jun 19, 2018 at 4:52 AM T Smith <ai...@gmail.com> wrote:
>
>> Hi Sandeep,
>>
>> I've had a look at this and here's what I think - maybe we should take
>> this to knox-developers, I'm happy to help work out a solution but I'm
>> going to need some insight into the design intent around this module.
>>
>> a) The existing unit test is hard-coded to test a backend URL that ends
>> in /ws. This corresponds to a special case in the code for Zeppelin, so the
>> other paths are never tested, and hence the unit test always passes.
>> b) createWebSocket passes only a path, so the query component could
>> never be considered by getMatchedBackendURL
>> c) getMatchedBackendURL doesn't seem to base its logic on the rewrite
>> rules at all, in the case where the backend URL doesn't end in /ws it
>> appends the remainder of the path to the backend (you can demonstrate this
>> by altering the test case to remove the /ws from the backend URL and
>> running the existing test - the rewrite rule is {**}/channels but the
>> result is simply /channels).
>> d) Due to (a) this is a moot point, but the unit test doesn't check that
>> the path was rewritten as expected, so it will pass regardless.
>>
>> I think if we address (b) to pass a concatenation of the path and query
>> then it might work for cases like mine, but this doesn't address (c) which
>> will affect anyone wanting to control rewrites. I guess addressing (a) and
>> (d) would help in the longer term.
>>
>> Hope this makes sense and I haven't completely missed the point :-) It
>> does explain the behaviour I'm seeing. How would you like to proceed here?
>>
>> Cheers,
>> /ailuropod4
>>
>>
>>
>>
>> On Mon, Jun 18, 2018 at 7:26 PM, Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> may be you are right about not not properly handeling the query params
>>> and we might need to fix this regex:
>>>
>>> final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
>>>
>>> You should be able to do a quick test using a this Unit Test
>>> <https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
>>>
>>> Best,
>>> Sandeep
>>>
>>>
>>>
>>>
>>> On Mon, Jun 18, 2018 at 1:55 PM T Smith <ai...@gmail.com> wrote:
>>>
>>>> Sorry, bad example.
>>>>
>>>> I send ws://<host>:<port>/gateway/pnda/pndaconsole/
>>>> socket.io/?EIO=3&transport=websocket
>>>>
>>>> Knox sends /socket.io/
>>>>
>>>> That yields a 400 error, Knox throws
>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>>
>>>>
>>>> On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ai...@gmail.com> wrote:
>>>>
>>>>> Hi Sandeep,
>>>>>
>>>>> Back to fighting with this. Through some nginx debugging on my backend
>>>>> I've come to the conclusion that Knox isn't sending the query parameters on
>>>>> to the backend, regardless of what rewrite rules I specify.
>>>>>
>>>>> I.e. I send
>>>>> /gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>
>>>>> Knox sends /socket.io/
>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>
>>>>> This yields a 400 error, and Knox drops the websocket to the backend.
>>>>> What was confusing me was the 101 in the browser, but I see now that Knox
>>>>> is terminating the connection with the browser on one side and opening
>>>>> another connection with the backend on the other.
>>>>>
>>>>> I see this
>>>>> https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc but I'm
>>>>> wondering if this code needs further work to support query parameters? What
>>>>> do you think?
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> /ailuropod4
>>>>>
>>>>> On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com> wrote:
>>>>>
>>>>>> Hi Sandeep,
>>>>>>
>>>>>> Looking at my nginx server, it never sees the transport=websocket
>>>>>> ws:// protocol request, but it does see a http request for / at the
>>>>>> corresponding time, so I think perhaps the root problem here is in that
>>>>>> rewrite, as you mention. By the way, the reason for ws:// and not wss:// is
>>>>>> I switched off TLS to remove another potential source of issue, so it's all
>>>>>> plain http. I saw the same behaviour over https.
>>>>>>
>>>>>> My services, now, look like this -
>>>>>>
>>>>>> <route path="/pndaconsole/socket.io">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>> to="request.url"/>
>>>>>> </route>
>>>>>>
>>>>>> <route path="/pndaconsole/metrics">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>>> to="request.url"/>
>>>>>> </route>
>>>>>>
>>>>>> <route path="/pndaconsole/">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>> to="request.url"/>
>>>>>> </route>
>>>>>>
>>>>>> <route path="/pndaconsole/**">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>> to="request.url"/>
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>> to="response.body"/>
>>>>>> </route>
>>>>>>
>>>>>> My inbound rewrites, now, look like this -
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>> </rule>
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>>> </rule>
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>> </rule>
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>>> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/>
>>>>>> </rule>
>>>>>>
>>>>>> The ws://...transport=websocket requests are rewritten as /, so I
>>>>>> tried adding an entry in my topology for WEBSOCKET and an additional
>>>>>> inbound rule like this -
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>>> </rule>
>>>>>>
>>>>>> Where the topology entry is the same as PNDACONSOLE except it uses
>>>>>> the ws:// protocol specifier. This didn't work at all, and in fact at that
>>>>>> point, nothing got through and everything fell through to the default
>>>>>> rewrite rule.
>>>>>>
>>>>>> I've attached the gateway log with debug turned on everywhere
>>>>>> (tarballed as it's huge).
>>>>>>
>>>>>> I guess what I'm missing is a clear idea of what I'm supposed to do
>>>>>> in the service/rewrite rules to deal with the websockets request which will
>>>>>> arrive with a different protocol specifier and a different query parameter.
>>>>>> Should I explicitly map those through to the back end, or is this something
>>>>>> Knox handles internally?
>>>>>>
>>>>>> Any insight into this appreciated, it seems very close to working,
>>>>>> and in fact socket.io does it's usual fallback so it is functional
>>>>>> after a fashion but it seems like this should fully work given what I see
>>>>>> with Zeppelin.
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> /ailuropod4
>>>>>>
>>>>>>
>>>>>> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <mo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello ailuropod4
>>>>>>>
>>>>>>> You should not have to reroute to a different host port, this should
>>>>>>> work.
>>>>>>>
>>>>>>> Looking at the responses, it looks like protocol switching happens
>>>>>>> at the Knox end, between browser and Knox (going by the url ws://
>>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW)
>>>>>>> but it should have
>>>>>>> been wss:// wonder why this is ws://.
>>>>>>>
>>>>>>> Also, can you include seperate "ws" rules for your service, similar
>>>>>>> to Zeppelin.
>>>>>>> Also, make sure you have, inbound routes in service.xml for e.g.
>>>>>>>
>>>>>>> <routes>
>>>>>>> <route path="/zeppelin/ws">
>>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound"
>>>>>>> to="request.url"/>
>>>>>>> </route>
>>>>>>>
>>>>>>> <route path="/zeppelin/ws**">
>>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/>
>>>>>>> </route>
>>>>>>> </routes>
>>>>>>>
>>>>>>> and then reference it in rewrite.xml.
>>>>>>>
>>>>>>> Looking at the error, it looks like the issue is not with rewrite
>>>>>>> rules but with the connection between Knox and the backend.
>>>>>>> We can see that Knox is initiating an upgrade request but then for
>>>>>>> some reason the backend service is closing/refusing it.
>>>>>>> Can you turn up the logging in Knox to Debug and share it if you can
>>>>>>> (this should atleast tell us whether rewrite rules are working).
>>>>>>>
>>>>>>> Also, do you see any errors at the backend service ? it woud be good
>>>>>>> to know what the backend is seeing and reasons for it to refuse the upgrade
>>>>>>> and also the error code
>>>>>>> Unfortunately the error code is gobbled up somewhere in the process.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> And if it helps, the stack trace as a result of this from Knox
>>>>>>>> follows.
>>>>>>>>
>>>>>>>> Can I match a url containing &transport=websocket in one of the
>>>>>>>> query parameters and map it through to a service defined as ws:// in the
>>>>>>>> topology? At the moment, as above, everything is sent to a http://
>>>>>>>> service, maybe this is the crux of the problem?
>>>>>>>>
>>>>>>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>>>>>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>>>>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>>>>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>>>>>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to connect to
>>>>>>>> websocket server: java.io.IOException: Connect failure
>>>>>>>> java.io.IOException: Connect failure
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>>>>> at
>>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>>> at org.eclipse.jetty.websocket.se
>>>>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>>>> at org.eclipse.jetty.io
>>>>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>>>> at org.eclipse.jetty.io
>>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>>>> switch protocols
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>>> ... 4 more
>>>>>>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>>>>>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing
>>>>>>>> connection)
>>>>>>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException:
>>>>>>>> Connect failure
>>>>>>>> at
>>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:106)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>>> at org.eclipse.jetty.websocket.se
>>>>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>>>> at org.eclipse.jetty.io
>>>>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>>>> at org.eclipse.jetty.io
>>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: java.io.IOException: Connect failure
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>>>>> at
>>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>>>> ... 12 more
>>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>>>> switch protocols
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>>> ... 4 more
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>>>>>>
>>>>>>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Sandeep,
>>>>>>>>>>
>>>>>>>>>> Here's what's happening -
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1. Request URL:
>>>>>>>>>>
>>>>>>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>>>>>>> 2. Request Method:
>>>>>>>>>> GET
>>>>>>>>>> 3. Status Code:
>>>>>>>>>> 200 OK
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1. Request URL:
>>>>>>>>>> ws://
>>>>>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>>>>>>> 2. Request Method:
>>>>>>>>>> GET
>>>>>>>>>> 3. Status Code:
>>>>>>>>>> 101 Switching Protocols
>>>>>>>>>>
>>>>>>>>>> So far, so good. But, the next couple of requests fail, with 400
>>>>>>>>>> or 500 range errors, and the gateway log complains about protocol errors. I
>>>>>>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>>>>>>> big picture across.
>>>>>>>>>>
>>>>>>>>>> Looking at tcpdump, what's happening is that ws:// request is
>>>>>>>>>> being nerfed to a GET /, which on my backend happens to resolve to a nice
>>>>>>>>>> front page of HTML. Websockets obviously panics as this isn't what it
>>>>>>>>>> expected, and then the whole process repeats. On my backend, /
>>>>>>>>>> socket.io is routed to the socket endpoint, whereas / isn't. So,
>>>>>>>>>> my theory is that if I can somehow convince Knox to send out the ws://
>>>>>>>>>> request with the correct path including /socket.io, it might
>>>>>>>>>> work.
>>>>>>>>>>
>>>>>>>>>> At the moment my service/rewrite sections are very simple, I pick
>>>>>>>>>> up requests with socket.io and send them off to the backend.
>>>>>>>>>>
>>>>>>>>>> services.xml -
>>>>>>>>>>
>>>>>>>>>> <policies>
>>>>>>>>>> <policy role="webappsec"/>
>>>>>>>>>> <policy role="authentication" name="Anonymous"/>
>>>>>>>>>> <policy role="rewrite"/>
>>>>>>>>>> <policy role="authorization"/>
>>>>>>>>>> </policies>
>>>>>>>>>> <routes>
>>>>>>>>>> <route path="/pndaconsole">
>>>>>>>>>> </route>
>>>>>>>>>> <route path="/pndaconsole/**">
>>>>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>>>>>> to="response.body"/>
>>>>>>>>>> </route>
>>>>>>>>>> </routes>
>>>>>>>>>>
>>>>>>>>>> rewrite.xml -
>>>>>>>>>>
>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>>>>>> </rule>
>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/
>>>>>>>>>> socket.io/?{**} <http://socket.io/?%7B**%7D>"/>
>>>>>>>>>> </rule>
>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>>>>>> </rule>
>>>>>>>>>> (I've snipped the rest of the rules handling outbound rewrites as
>>>>>>>>>> I don't believe they're relevant, but let me know if you think otherwise)
>>>>>>>>>>
>>>>>>>>>> By all accounts it looks like it's *nearly* working, but I'm
>>>>>>>>>> stumped on where my /socket.io path is going. I was considering
>>>>>>>>>> routing all websockets stuff to a completely separate backend/port so that
>>>>>>>>>> it would go to the right place regardless of how it was rewritten, but that
>>>>>>>>>> seems a bit extreme and I was hoping to use this approach.
>>>>>>>>>>
>>>>>>>>>> My end to end is -
>>>>>>>>>>
>>>>>>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>>>>>>> websockets backend in nodejs
>>>>>>>>>>
>>>>>>>>>> --> flat files, being an angularjs app
>>>>>>>>>>
>>>>>>>>>> We don't have to use socket.io, I guess, but it is currently
>>>>>>>>>> used everywhere so I'd rather avoid swapping that out to get this working
>>>>>>>>>> if possible. It's a fairly common library.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <
>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I am not very familiar with socket.io apps, so you might have
>>>>>>>>>>> to fill me in about what you are trying to do.
>>>>>>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>>>>>>
>>>>>>>>>>> What version of Knox are you using ?
>>>>>>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has
>>>>>>>>>>> some examples in the comments.
>>>>>>>>>>>
>>>>>>>>>>> What is the ws:// url you are trying to rewrite and the rules
>>>>>>>>>>> you are using ?
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Sandeep
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm struggling to get a simple socket.io based application
>>>>>>>>>>>> working through Knox.
>>>>>>>>>>>>
>>>>>>>>>>>> I see websockets is supported, but the upgrade handshake seems
>>>>>>>>>>>> to be nerfed on the way out, specifically the /socket.io/? etc
>>>>>>>>>>>> part simply becomes /.
>>>>>>>>>>>>
>>>>>>>>>>>> I've tried lots of permutations - does anyone have a working
>>>>>>>>>>>> example they can point me towards? I've looked at the Zeppelin approach,
>>>>>>>>>>>> but this isn't quite the same thing (not socket.io).
>>>>>>>>>>>>
>>>>>>>>>>>> Any help appreciated!
>>>>>>>>>>>>
>>>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
Re: socket.io in knox
Posted by Sandeep Moré <mo...@gmail.com>.
I was thinking about the issue (c) I think the issue here is with the way
GatewayWebsocketHandler is added in GatewayServer.createhandler() class. I
am thinking the reason why the rewrite rules are skipped is because
GatewayWebsocketHandler
is called before other handlers, we need to test the order in which this
handler is added.
Best,
Sandeep
On Wed, Jun 20, 2018 at 1:13 AM Sandeep Moré <mo...@gmail.com> wrote:
> Hello ailuropod4,
>
> Excellent analysis !
> you are right about (a), (b), (c) and (d).
> (a) and (b) can be easily fixed, we should have caught this earlier :( we
> do need to account for url query string and url fragment, I am thinking, in
> the createWebSocket() method we can extract query and fragment along with
> path and concatenate it to path, we then use this path to pass it down to
> getMatchedBackendURL(), like you pointed out this should hopefully help
> you.
>
> It would be awesome if you could create a JIRA for this issue (if we do
> not have already) and create a patch along with a UnitTest, this would be
> extremely useful and should take care of (a) and (b)
>
> I think we need to look closely at (c), can you open a JIRA for (c) as
> well so we can investigate. I was hoping for the request to hit
> UrlRewriteServletFilter.doFilter() before reaching websocket handler,
> specifically hitting the following call
>
> UrlRewriteRequest rewriteRequest = new UrlRewriteRequest( config, request
> );
>
> From your observation it appears that this might not be the case which is
> a bug !
> I might not be able to look at it closely for few more days, it would be
> awesome if you could also take a look at it.
>
> Again thanks for pointing these issues :)
>
> Best,
> Sandeep
>
> On Tue, Jun 19, 2018 at 4:52 AM T Smith <ai...@gmail.com> wrote:
>
>> Hi Sandeep,
>>
>> I've had a look at this and here's what I think - maybe we should take
>> this to knox-developers, I'm happy to help work out a solution but I'm
>> going to need some insight into the design intent around this module.
>>
>> a) The existing unit test is hard-coded to test a backend URL that ends
>> in /ws. This corresponds to a special case in the code for Zeppelin, so the
>> other paths are never tested, and hence the unit test always passes.
>> b) createWebSocket passes only a path, so the query component could
>> never be considered by getMatchedBackendURL
>> c) getMatchedBackendURL doesn't seem to base its logic on the rewrite
>> rules at all, in the case where the backend URL doesn't end in /ws it
>> appends the remainder of the path to the backend (you can demonstrate this
>> by altering the test case to remove the /ws from the backend URL and
>> running the existing test - the rewrite rule is {**}/channels but the
>> result is simply /channels).
>> d) Due to (a) this is a moot point, but the unit test doesn't check that
>> the path was rewritten as expected, so it will pass regardless.
>>
>> I think if we address (b) to pass a concatenation of the path and query
>> then it might work for cases like mine, but this doesn't address (c) which
>> will affect anyone wanting to control rewrites. I guess addressing (a) and
>> (d) would help in the longer term.
>>
>> Hope this makes sense and I haven't completely missed the point :-) It
>> does explain the behaviour I'm seeing. How would you like to proceed here?
>>
>> Cheers,
>> /ailuropod4
>>
>>
>>
>>
>> On Mon, Jun 18, 2018 at 7:26 PM, Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> may be you are right about not not properly handeling the query params
>>> and we might need to fix this regex:
>>>
>>> final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
>>>
>>> You should be able to do a quick test using a this Unit Test
>>> <https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
>>>
>>> Best,
>>> Sandeep
>>>
>>>
>>>
>>>
>>> On Mon, Jun 18, 2018 at 1:55 PM T Smith <ai...@gmail.com> wrote:
>>>
>>>> Sorry, bad example.
>>>>
>>>> I send ws://<host>:<port>/gateway/pnda/pndaconsole/
>>>> socket.io/?EIO=3&transport=websocket
>>>>
>>>> Knox sends /socket.io/
>>>>
>>>> That yields a 400 error, Knox throws
>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>>
>>>>
>>>> On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ai...@gmail.com> wrote:
>>>>
>>>>> Hi Sandeep,
>>>>>
>>>>> Back to fighting with this. Through some nginx debugging on my backend
>>>>> I've come to the conclusion that Knox isn't sending the query parameters on
>>>>> to the backend, regardless of what rewrite rules I specify.
>>>>>
>>>>> I.e. I send
>>>>> /gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>
>>>>> Knox sends /socket.io/
>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>
>>>>> This yields a 400 error, and Knox drops the websocket to the backend.
>>>>> What was confusing me was the 101 in the browser, but I see now that Knox
>>>>> is terminating the connection with the browser on one side and opening
>>>>> another connection with the backend on the other.
>>>>>
>>>>> I see this
>>>>> https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc but I'm
>>>>> wondering if this code needs further work to support query parameters? What
>>>>> do you think?
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> /ailuropod4
>>>>>
>>>>> On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com> wrote:
>>>>>
>>>>>> Hi Sandeep,
>>>>>>
>>>>>> Looking at my nginx server, it never sees the transport=websocket
>>>>>> ws:// protocol request, but it does see a http request for / at the
>>>>>> corresponding time, so I think perhaps the root problem here is in that
>>>>>> rewrite, as you mention. By the way, the reason for ws:// and not wss:// is
>>>>>> I switched off TLS to remove another potential source of issue, so it's all
>>>>>> plain http. I saw the same behaviour over https.
>>>>>>
>>>>>> My services, now, look like this -
>>>>>>
>>>>>> <route path="/pndaconsole/socket.io">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>> to="request.url"/>
>>>>>> </route>
>>>>>>
>>>>>> <route path="/pndaconsole/metrics">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>>> to="request.url"/>
>>>>>> </route>
>>>>>>
>>>>>> <route path="/pndaconsole/">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>> to="request.url"/>
>>>>>> </route>
>>>>>>
>>>>>> <route path="/pndaconsole/**">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>> to="request.url"/>
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>> to="response.body"/>
>>>>>> </route>
>>>>>>
>>>>>> My inbound rewrites, now, look like this -
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>> </rule>
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>>> </rule>
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>> </rule>
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>>> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/>
>>>>>> </rule>
>>>>>>
>>>>>> The ws://...transport=websocket requests are rewritten as /, so I
>>>>>> tried adding an entry in my topology for WEBSOCKET and an additional
>>>>>> inbound rule like this -
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>>> </rule>
>>>>>>
>>>>>> Where the topology entry is the same as PNDACONSOLE except it uses
>>>>>> the ws:// protocol specifier. This didn't work at all, and in fact at that
>>>>>> point, nothing got through and everything fell through to the default
>>>>>> rewrite rule.
>>>>>>
>>>>>> I've attached the gateway log with debug turned on everywhere
>>>>>> (tarballed as it's huge).
>>>>>>
>>>>>> I guess what I'm missing is a clear idea of what I'm supposed to do
>>>>>> in the service/rewrite rules to deal with the websockets request which will
>>>>>> arrive with a different protocol specifier and a different query parameter.
>>>>>> Should I explicitly map those through to the back end, or is this something
>>>>>> Knox handles internally?
>>>>>>
>>>>>> Any insight into this appreciated, it seems very close to working,
>>>>>> and in fact socket.io does it's usual fallback so it is functional
>>>>>> after a fashion but it seems like this should fully work given what I see
>>>>>> with Zeppelin.
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> /ailuropod4
>>>>>>
>>>>>>
>>>>>> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <mo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello ailuropod4
>>>>>>>
>>>>>>> You should not have to reroute to a different host port, this should
>>>>>>> work.
>>>>>>>
>>>>>>> Looking at the responses, it looks like protocol switching happens
>>>>>>> at the Knox end, between browser and Knox (going by the url ws://
>>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW)
>>>>>>> but it should have
>>>>>>> been wss:// wonder why this is ws://.
>>>>>>>
>>>>>>> Also, can you include seperate "ws" rules for your service, similar
>>>>>>> to Zeppelin.
>>>>>>> Also, make sure you have, inbound routes in service.xml for e.g.
>>>>>>>
>>>>>>> <routes>
>>>>>>> <route path="/zeppelin/ws">
>>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound"
>>>>>>> to="request.url"/>
>>>>>>> </route>
>>>>>>>
>>>>>>> <route path="/zeppelin/ws**">
>>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/>
>>>>>>> </route>
>>>>>>> </routes>
>>>>>>>
>>>>>>> and then reference it in rewrite.xml.
>>>>>>>
>>>>>>> Looking at the error, it looks like the issue is not with rewrite
>>>>>>> rules but with the connection between Knox and the backend.
>>>>>>> We can see that Knox is initiating an upgrade request but then for
>>>>>>> some reason the backend service is closing/refusing it.
>>>>>>> Can you turn up the logging in Knox to Debug and share it if you can
>>>>>>> (this should atleast tell us whether rewrite rules are working).
>>>>>>>
>>>>>>> Also, do you see any errors at the backend service ? it woud be good
>>>>>>> to know what the backend is seeing and reasons for it to refuse the upgrade
>>>>>>> and also the error code
>>>>>>> Unfortunately the error code is gobbled up somewhere in the process.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> And if it helps, the stack trace as a result of this from Knox
>>>>>>>> follows.
>>>>>>>>
>>>>>>>> Can I match a url containing &transport=websocket in one of the
>>>>>>>> query parameters and map it through to a service defined as ws:// in the
>>>>>>>> topology? At the moment, as above, everything is sent to a http://
>>>>>>>> service, maybe this is the crux of the problem?
>>>>>>>>
>>>>>>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>>>>>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>>>>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>>>>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>>>>>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to connect to
>>>>>>>> websocket server: java.io.IOException: Connect failure
>>>>>>>> java.io.IOException: Connect failure
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>>>>> at
>>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>>> at org.eclipse.jetty.websocket.se
>>>>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>>>> at org.eclipse.jetty.io
>>>>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>>>> at org.eclipse.jetty.io
>>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>>>> switch protocols
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>>> ... 4 more
>>>>>>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>>>>>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing
>>>>>>>> connection)
>>>>>>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException:
>>>>>>>> Connect failure
>>>>>>>> at
>>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:106)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>>> at org.eclipse.jetty.websocket.se
>>>>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>>>> at org.eclipse.jetty.io
>>>>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>>>> at org.eclipse.jetty.io
>>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: java.io.IOException: Connect failure
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>>>>> at
>>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>>>>> at
>>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>>>> ... 12 more
>>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>>>> switch protocols
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>>> ... 4 more
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>>>>>>
>>>>>>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Sandeep,
>>>>>>>>>>
>>>>>>>>>> Here's what's happening -
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1. Request URL:
>>>>>>>>>>
>>>>>>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>>>>>>> 2. Request Method:
>>>>>>>>>> GET
>>>>>>>>>> 3. Status Code:
>>>>>>>>>> 200 OK
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1. Request URL:
>>>>>>>>>> ws://
>>>>>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>>>>>>> 2. Request Method:
>>>>>>>>>> GET
>>>>>>>>>> 3. Status Code:
>>>>>>>>>> 101 Switching Protocols
>>>>>>>>>>
>>>>>>>>>> So far, so good. But, the next couple of requests fail, with 400
>>>>>>>>>> or 500 range errors, and the gateway log complains about protocol errors. I
>>>>>>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>>>>>>> big picture across.
>>>>>>>>>>
>>>>>>>>>> Looking at tcpdump, what's happening is that ws:// request is
>>>>>>>>>> being nerfed to a GET /, which on my backend happens to resolve to a nice
>>>>>>>>>> front page of HTML. Websockets obviously panics as this isn't what it
>>>>>>>>>> expected, and then the whole process repeats. On my backend, /
>>>>>>>>>> socket.io is routed to the socket endpoint, whereas / isn't. So,
>>>>>>>>>> my theory is that if I can somehow convince Knox to send out the ws://
>>>>>>>>>> request with the correct path including /socket.io, it might
>>>>>>>>>> work.
>>>>>>>>>>
>>>>>>>>>> At the moment my service/rewrite sections are very simple, I pick
>>>>>>>>>> up requests with socket.io and send them off to the backend.
>>>>>>>>>>
>>>>>>>>>> services.xml -
>>>>>>>>>>
>>>>>>>>>> <policies>
>>>>>>>>>> <policy role="webappsec"/>
>>>>>>>>>> <policy role="authentication" name="Anonymous"/>
>>>>>>>>>> <policy role="rewrite"/>
>>>>>>>>>> <policy role="authorization"/>
>>>>>>>>>> </policies>
>>>>>>>>>> <routes>
>>>>>>>>>> <route path="/pndaconsole">
>>>>>>>>>> </route>
>>>>>>>>>> <route path="/pndaconsole/**">
>>>>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>>>>>> to="response.body"/>
>>>>>>>>>> </route>
>>>>>>>>>> </routes>
>>>>>>>>>>
>>>>>>>>>> rewrite.xml -
>>>>>>>>>>
>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>>>>>> </rule>
>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/
>>>>>>>>>> socket.io/?{**} <http://socket.io/?%7B**%7D>"/>
>>>>>>>>>> </rule>
>>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>>>>>> </rule>
>>>>>>>>>> (I've snipped the rest of the rules handling outbound rewrites as
>>>>>>>>>> I don't believe they're relevant, but let me know if you think otherwise)
>>>>>>>>>>
>>>>>>>>>> By all accounts it looks like it's *nearly* working, but I'm
>>>>>>>>>> stumped on where my /socket.io path is going. I was considering
>>>>>>>>>> routing all websockets stuff to a completely separate backend/port so that
>>>>>>>>>> it would go to the right place regardless of how it was rewritten, but that
>>>>>>>>>> seems a bit extreme and I was hoping to use this approach.
>>>>>>>>>>
>>>>>>>>>> My end to end is -
>>>>>>>>>>
>>>>>>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>>>>>>> websockets backend in nodejs
>>>>>>>>>>
>>>>>>>>>> --> flat files, being an angularjs app
>>>>>>>>>>
>>>>>>>>>> We don't have to use socket.io, I guess, but it is currently
>>>>>>>>>> used everywhere so I'd rather avoid swapping that out to get this working
>>>>>>>>>> if possible. It's a fairly common library.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <
>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I am not very familiar with socket.io apps, so you might have
>>>>>>>>>>> to fill me in about what you are trying to do.
>>>>>>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>>>>>>
>>>>>>>>>>> What version of Knox are you using ?
>>>>>>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has
>>>>>>>>>>> some examples in the comments.
>>>>>>>>>>>
>>>>>>>>>>> What is the ws:// url you are trying to rewrite and the rules
>>>>>>>>>>> you are using ?
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Sandeep
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm struggling to get a simple socket.io based application
>>>>>>>>>>>> working through Knox.
>>>>>>>>>>>>
>>>>>>>>>>>> I see websockets is supported, but the upgrade handshake seems
>>>>>>>>>>>> to be nerfed on the way out, specifically the /socket.io/? etc
>>>>>>>>>>>> part simply becomes /.
>>>>>>>>>>>>
>>>>>>>>>>>> I've tried lots of permutations - does anyone have a working
>>>>>>>>>>>> example they can point me towards? I've looked at the Zeppelin approach,
>>>>>>>>>>>> but this isn't quite the same thing (not socket.io).
>>>>>>>>>>>>
>>>>>>>>>>>> Any help appreciated!
>>>>>>>>>>>>
>>>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
Re: socket.io in knox
Posted by Sandeep Moré <mo...@gmail.com>.
Hello ailuropod4,
Excellent analysis !
you are right about (a), (b), (c) and (d).
(a) and (b) can be easily fixed, we should have caught this earlier :( we
do need to account for url query string and url fragment, I am thinking, in
the createWebSocket() method we can extract query and fragment along with
path and concatenate it to path, we then use this path to pass it down to
getMatchedBackendURL(), like you pointed out this should hopefully help
you.
It would be awesome if you could create a JIRA for this issue (if we do not
have already) and create a patch along with a UnitTest, this would be
extremely useful and should take care of (a) and (b)
I think we need to look closely at (c), can you open a JIRA for (c) as well
so we can investigate. I was hoping for the request to hit
UrlRewriteServletFilter.doFilter() before reaching websocket handler,
specifically hitting the following call
UrlRewriteRequest rewriteRequest = new UrlRewriteRequest( config, request );
From your observation it appears that this might not be the case which is a
bug !
I might not be able to look at it closely for few more days, it would be
awesome if you could also take a look at it.
Again thanks for pointing these issues :)
Best,
Sandeep
On Tue, Jun 19, 2018 at 4:52 AM T Smith <ai...@gmail.com> wrote:
> Hi Sandeep,
>
> I've had a look at this and here's what I think - maybe we should take
> this to knox-developers, I'm happy to help work out a solution but I'm
> going to need some insight into the design intent around this module.
>
> a) The existing unit test is hard-coded to test a backend URL that ends in
> /ws. This corresponds to a special case in the code for Zeppelin, so the
> other paths are never tested, and hence the unit test always passes.
> b) createWebSocket passes only a path, so the query component could never
> be considered by getMatchedBackendURL
> c) getMatchedBackendURL doesn't seem to base its logic on the rewrite
> rules at all, in the case where the backend URL doesn't end in /ws it
> appends the remainder of the path to the backend (you can demonstrate this
> by altering the test case to remove the /ws from the backend URL and
> running the existing test - the rewrite rule is {**}/channels but the
> result is simply /channels).
> d) Due to (a) this is a moot point, but the unit test doesn't check that
> the path was rewritten as expected, so it will pass regardless.
>
> I think if we address (b) to pass a concatenation of the path and query
> then it might work for cases like mine, but this doesn't address (c) which
> will affect anyone wanting to control rewrites. I guess addressing (a) and
> (d) would help in the longer term.
>
> Hope this makes sense and I haven't completely missed the point :-) It
> does explain the behaviour I'm seeing. How would you like to proceed here?
>
> Cheers,
> /ailuropod4
>
>
>
>
> On Mon, Jun 18, 2018 at 7:26 PM, Sandeep Moré <mo...@gmail.com>
> wrote:
>
>> may be you are right about not not properly handeling the query params
>> and we might need to fix this regex:
>>
>> final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
>>
>> You should be able to do a quick test using a this Unit Test
>> <https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
>>
>> Best,
>> Sandeep
>>
>>
>>
>>
>> On Mon, Jun 18, 2018 at 1:55 PM T Smith <ai...@gmail.com> wrote:
>>
>>> Sorry, bad example.
>>>
>>> I send ws://<host>:<port>/gateway/pnda/pndaconsole/
>>> socket.io/?EIO=3&transport=websocket
>>>
>>> Knox sends /socket.io/
>>>
>>> That yields a 400 error, Knox throws
>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>
>>>
>>> On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ai...@gmail.com> wrote:
>>>
>>>> Hi Sandeep,
>>>>
>>>> Back to fighting with this. Through some nginx debugging on my backend
>>>> I've come to the conclusion that Knox isn't sending the query parameters on
>>>> to the backend, regardless of what rewrite rules I specify.
>>>>
>>>> I.e. I send
>>>> /gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>
>>>> Knox sends /socket.io/
>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>
>>>> This yields a 400 error, and Knox drops the websocket to the backend.
>>>> What was confusing me was the 101 in the browser, but I see now that Knox
>>>> is terminating the connection with the browser on one side and opening
>>>> another connection with the backend on the other.
>>>>
>>>> I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc
>>>> but I'm wondering if this code needs further work to support query
>>>> parameters? What do you think?
>>>>
>>>> Thanks in advance!
>>>>
>>>> /ailuropod4
>>>>
>>>> On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com> wrote:
>>>>
>>>>> Hi Sandeep,
>>>>>
>>>>> Looking at my nginx server, it never sees the transport=websocket
>>>>> ws:// protocol request, but it does see a http request for / at the
>>>>> corresponding time, so I think perhaps the root problem here is in that
>>>>> rewrite, as you mention. By the way, the reason for ws:// and not wss:// is
>>>>> I switched off TLS to remove another potential source of issue, so it's all
>>>>> plain http. I saw the same behaviour over https.
>>>>>
>>>>> My services, now, look like this -
>>>>>
>>>>> <route path="/pndaconsole/socket.io">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>> to="request.url"/>
>>>>> </route>
>>>>>
>>>>> <route path="/pndaconsole/metrics">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>> to="request.url"/>
>>>>> </route>
>>>>>
>>>>> <route path="/pndaconsole/">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
>>>>> to="request.url"/>
>>>>> </route>
>>>>>
>>>>> <route path="/pndaconsole/**">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
>>>>> to="request.url"/>
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>> to="response.body"/>
>>>>> </route>
>>>>>
>>>>> My inbound rewrites, now, look like this -
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>> </rule>
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>> </rule>
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>> </rule>
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/>
>>>>> </rule>
>>>>>
>>>>> The ws://...transport=websocket requests are rewritten as /, so I
>>>>> tried adding an entry in my topology for WEBSOCKET and an additional
>>>>> inbound rule like this -
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>">
>>>>> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>> </rule>
>>>>>
>>>>> Where the topology entry is the same as PNDACONSOLE except it uses the
>>>>> ws:// protocol specifier. This didn't work at all, and in fact at that
>>>>> point, nothing got through and everything fell through to the default
>>>>> rewrite rule.
>>>>>
>>>>> I've attached the gateway log with debug turned on everywhere
>>>>> (tarballed as it's huge).
>>>>>
>>>>> I guess what I'm missing is a clear idea of what I'm supposed to do in
>>>>> the service/rewrite rules to deal with the websockets request which will
>>>>> arrive with a different protocol specifier and a different query parameter.
>>>>> Should I explicitly map those through to the back end, or is this something
>>>>> Knox handles internally?
>>>>>
>>>>> Any insight into this appreciated, it seems very close to working, and
>>>>> in fact socket.io does it's usual fallback so it is functional after
>>>>> a fashion but it seems like this should fully work given what I see with
>>>>> Zeppelin.
>>>>>
>>>>>
>>>>> Cheers,
>>>>> /ailuropod4
>>>>>
>>>>>
>>>>> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello ailuropod4
>>>>>>
>>>>>> You should not have to reroute to a different host port, this should
>>>>>> work.
>>>>>>
>>>>>> Looking at the responses, it looks like protocol switching happens at
>>>>>> the Knox end, between browser and Knox (going by the url ws://
>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW)
>>>>>> but it should have
>>>>>> been wss:// wonder why this is ws://.
>>>>>>
>>>>>> Also, can you include seperate "ws" rules for your service, similar
>>>>>> to Zeppelin.
>>>>>> Also, make sure you have, inbound routes in service.xml for e.g.
>>>>>>
>>>>>> <routes>
>>>>>> <route path="/zeppelin/ws">
>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound"
>>>>>> to="request.url"/>
>>>>>> </route>
>>>>>>
>>>>>> <route path="/zeppelin/ws**">
>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/>
>>>>>> </route>
>>>>>> </routes>
>>>>>>
>>>>>> and then reference it in rewrite.xml.
>>>>>>
>>>>>> Looking at the error, it looks like the issue is not with rewrite
>>>>>> rules but with the connection between Knox and the backend.
>>>>>> We can see that Knox is initiating an upgrade request but then for
>>>>>> some reason the backend service is closing/refusing it.
>>>>>> Can you turn up the logging in Knox to Debug and share it if you can
>>>>>> (this should atleast tell us whether rewrite rules are working).
>>>>>>
>>>>>> Also, do you see any errors at the backend service ? it woud be good
>>>>>> to know what the backend is seeing and reasons for it to refuse the upgrade
>>>>>> and also the error code
>>>>>> Unfortunately the error code is gobbled up somewhere in the process.
>>>>>>
>>>>>> Best,
>>>>>> Sandeep
>>>>>>
>>>>>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> And if it helps, the stack trace as a result of this from Knox
>>>>>>> follows.
>>>>>>>
>>>>>>> Can I match a url containing &transport=websocket in one of the
>>>>>>> query parameters and map it through to a service defined as ws:// in the
>>>>>>> topology? At the moment, as above, everything is sent to a http://
>>>>>>> service, maybe this is the crux of the problem?
>>>>>>>
>>>>>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>>>>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>>>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>>>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>>>>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to connect to
>>>>>>> websocket server: java.io.IOException: Connect failure
>>>>>>> java.io.IOException: Connect failure
>>>>>>> at
>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>>>> at
>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>>>> at
>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>> at org.eclipse.jetty.websocket.se
>>>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>>> at org.eclipse.jetty.io
>>>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>>> at org.eclipse.jetty.io
>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>>>> at
>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>>> at
>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>>> switch protocols
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>> ... 4 more
>>>>>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>>>>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing
>>>>>>> connection)
>>>>>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException:
>>>>>>> Connect failure
>>>>>>> at
>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:106)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>> at org.eclipse.jetty.websocket.se
>>>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>>> at org.eclipse.jetty.io
>>>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>>> at org.eclipse.jetty.io
>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>>>> at
>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>>> at
>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: java.io.IOException: Connect failure
>>>>>>> at
>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>>>> at
>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>>>> at
>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>>> ... 12 more
>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>>> switch protocols
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>> ... 4 more
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>>>>>
>>>>>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Sandeep,
>>>>>>>>>
>>>>>>>>> Here's what's happening -
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1. Request URL:
>>>>>>>>>
>>>>>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>>>>>> 2. Request Method:
>>>>>>>>> GET
>>>>>>>>> 3. Status Code:
>>>>>>>>> 200 OK
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1. Request URL:
>>>>>>>>> ws://
>>>>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>>>>>> 2. Request Method:
>>>>>>>>> GET
>>>>>>>>> 3. Status Code:
>>>>>>>>> 101 Switching Protocols
>>>>>>>>>
>>>>>>>>> So far, so good. But, the next couple of requests fail, with 400
>>>>>>>>> or 500 range errors, and the gateway log complains about protocol errors. I
>>>>>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>>>>>> big picture across.
>>>>>>>>>
>>>>>>>>> Looking at tcpdump, what's happening is that ws:// request is
>>>>>>>>> being nerfed to a GET /, which on my backend happens to resolve to a nice
>>>>>>>>> front page of HTML. Websockets obviously panics as this isn't what it
>>>>>>>>> expected, and then the whole process repeats. On my backend, /
>>>>>>>>> socket.io is routed to the socket endpoint, whereas / isn't. So,
>>>>>>>>> my theory is that if I can somehow convince Knox to send out the ws://
>>>>>>>>> request with the correct path including /socket.io, it might work.
>>>>>>>>>
>>>>>>>>> At the moment my service/rewrite sections are very simple, I pick
>>>>>>>>> up requests with socket.io and send them off to the backend.
>>>>>>>>>
>>>>>>>>> services.xml -
>>>>>>>>>
>>>>>>>>> <policies>
>>>>>>>>> <policy role="webappsec"/>
>>>>>>>>> <policy role="authentication" name="Anonymous"/>
>>>>>>>>> <policy role="rewrite"/>
>>>>>>>>> <policy role="authorization"/>
>>>>>>>>> </policies>
>>>>>>>>> <routes>
>>>>>>>>> <route path="/pndaconsole">
>>>>>>>>> </route>
>>>>>>>>> <route path="/pndaconsole/**">
>>>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>>>>> to="response.body"/>
>>>>>>>>> </route>
>>>>>>>>> </routes>
>>>>>>>>>
>>>>>>>>> rewrite.xml -
>>>>>>>>>
>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>>>>> </rule>
>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/
>>>>>>>>> socket.io/?{**} <http://socket.io/?%7B**%7D>"/>
>>>>>>>>> </rule>
>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>>>>> </rule>
>>>>>>>>> (I've snipped the rest of the rules handling outbound rewrites as
>>>>>>>>> I don't believe they're relevant, but let me know if you think otherwise)
>>>>>>>>>
>>>>>>>>> By all accounts it looks like it's *nearly* working, but I'm
>>>>>>>>> stumped on where my /socket.io path is going. I was considering
>>>>>>>>> routing all websockets stuff to a completely separate backend/port so that
>>>>>>>>> it would go to the right place regardless of how it was rewritten, but that
>>>>>>>>> seems a bit extreme and I was hoping to use this approach.
>>>>>>>>>
>>>>>>>>> My end to end is -
>>>>>>>>>
>>>>>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>>>>>> websockets backend in nodejs
>>>>>>>>>
>>>>>>>>> --> flat files, being an angularjs app
>>>>>>>>>
>>>>>>>>> We don't have to use socket.io, I guess, but it is currently used
>>>>>>>>> everywhere so I'd rather avoid swapping that out to get this working if
>>>>>>>>> possible. It's a fairly common library.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <
>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I am not very familiar with socket.io apps, so you might have to
>>>>>>>>>> fill me in about what you are trying to do.
>>>>>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>>>>>
>>>>>>>>>> What version of Knox are you using ?
>>>>>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has
>>>>>>>>>> some examples in the comments.
>>>>>>>>>>
>>>>>>>>>> What is the ws:// url you are trying to rewrite and the rules you
>>>>>>>>>> are using ?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Sandeep
>>>>>>>>>>
>>>>>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I'm struggling to get a simple socket.io based application
>>>>>>>>>>> working through Knox.
>>>>>>>>>>>
>>>>>>>>>>> I see websockets is supported, but the upgrade handshake seems
>>>>>>>>>>> to be nerfed on the way out, specifically the /socket.io/? etc
>>>>>>>>>>> part simply becomes /.
>>>>>>>>>>>
>>>>>>>>>>> I've tried lots of permutations - does anyone have a working
>>>>>>>>>>> example they can point me towards? I've looked at the Zeppelin approach,
>>>>>>>>>>> but this isn't quite the same thing (not socket.io).
>>>>>>>>>>>
>>>>>>>>>>> Any help appreciated!
>>>>>>>>>>>
>>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: socket.io in knox
Posted by Sandeep Moré <mo...@gmail.com>.
Hello ailuropod4,
Excellent analysis !
you are right about (a), (b), (c) and (d).
(a) and (b) can be easily fixed, we should have caught this earlier :( we
do need to account for url query string and url fragment, I am thinking, in
the createWebSocket() method we can extract query and fragment along with
path and concatenate it to path, we then use this path to pass it down to
getMatchedBackendURL(), like you pointed out this should hopefully help
you.
It would be awesome if you could create a JIRA for this issue (if we do not
have already) and create a patch along with a UnitTest, this would be
extremely useful and should take care of (a) and (b)
I think we need to look closely at (c), can you open a JIRA for (c) as well
so we can investigate. I was hoping for the request to hit
UrlRewriteServletFilter.doFilter() before reaching websocket handler,
specifically hitting the following call
UrlRewriteRequest rewriteRequest = new UrlRewriteRequest( config, request );
From your observation it appears that this might not be the case which is a
bug !
I might not be able to look at it closely for few more days, it would be
awesome if you could also take a look at it.
Again thanks for pointing these issues :)
Best,
Sandeep
On Tue, Jun 19, 2018 at 4:52 AM T Smith <ai...@gmail.com> wrote:
> Hi Sandeep,
>
> I've had a look at this and here's what I think - maybe we should take
> this to knox-developers, I'm happy to help work out a solution but I'm
> going to need some insight into the design intent around this module.
>
> a) The existing unit test is hard-coded to test a backend URL that ends in
> /ws. This corresponds to a special case in the code for Zeppelin, so the
> other paths are never tested, and hence the unit test always passes.
> b) createWebSocket passes only a path, so the query component could never
> be considered by getMatchedBackendURL
> c) getMatchedBackendURL doesn't seem to base its logic on the rewrite
> rules at all, in the case where the backend URL doesn't end in /ws it
> appends the remainder of the path to the backend (you can demonstrate this
> by altering the test case to remove the /ws from the backend URL and
> running the existing test - the rewrite rule is {**}/channels but the
> result is simply /channels).
> d) Due to (a) this is a moot point, but the unit test doesn't check that
> the path was rewritten as expected, so it will pass regardless.
>
> I think if we address (b) to pass a concatenation of the path and query
> then it might work for cases like mine, but this doesn't address (c) which
> will affect anyone wanting to control rewrites. I guess addressing (a) and
> (d) would help in the longer term.
>
> Hope this makes sense and I haven't completely missed the point :-) It
> does explain the behaviour I'm seeing. How would you like to proceed here?
>
> Cheers,
> /ailuropod4
>
>
>
>
> On Mon, Jun 18, 2018 at 7:26 PM, Sandeep Moré <mo...@gmail.com>
> wrote:
>
>> may be you are right about not not properly handeling the query params
>> and we might need to fix this regex:
>>
>> final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
>>
>> You should be able to do a quick test using a this Unit Test
>> <https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
>>
>> Best,
>> Sandeep
>>
>>
>>
>>
>> On Mon, Jun 18, 2018 at 1:55 PM T Smith <ai...@gmail.com> wrote:
>>
>>> Sorry, bad example.
>>>
>>> I send ws://<host>:<port>/gateway/pnda/pndaconsole/
>>> socket.io/?EIO=3&transport=websocket
>>>
>>> Knox sends /socket.io/
>>>
>>> That yields a 400 error, Knox throws
>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>
>>>
>>> On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ai...@gmail.com> wrote:
>>>
>>>> Hi Sandeep,
>>>>
>>>> Back to fighting with this. Through some nginx debugging on my backend
>>>> I've come to the conclusion that Knox isn't sending the query parameters on
>>>> to the backend, regardless of what rewrite rules I specify.
>>>>
>>>> I.e. I send
>>>> /gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>
>>>> Knox sends /socket.io/
>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>
>>>> This yields a 400 error, and Knox drops the websocket to the backend.
>>>> What was confusing me was the 101 in the browser, but I see now that Knox
>>>> is terminating the connection with the browser on one side and opening
>>>> another connection with the backend on the other.
>>>>
>>>> I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc
>>>> but I'm wondering if this code needs further work to support query
>>>> parameters? What do you think?
>>>>
>>>> Thanks in advance!
>>>>
>>>> /ailuropod4
>>>>
>>>> On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com> wrote:
>>>>
>>>>> Hi Sandeep,
>>>>>
>>>>> Looking at my nginx server, it never sees the transport=websocket
>>>>> ws:// protocol request, but it does see a http request for / at the
>>>>> corresponding time, so I think perhaps the root problem here is in that
>>>>> rewrite, as you mention. By the way, the reason for ws:// and not wss:// is
>>>>> I switched off TLS to remove another potential source of issue, so it's all
>>>>> plain http. I saw the same behaviour over https.
>>>>>
>>>>> My services, now, look like this -
>>>>>
>>>>> <route path="/pndaconsole/socket.io">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>> to="request.url"/>
>>>>> </route>
>>>>>
>>>>> <route path="/pndaconsole/metrics">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>> to="request.url"/>
>>>>> </route>
>>>>>
>>>>> <route path="/pndaconsole/">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
>>>>> to="request.url"/>
>>>>> </route>
>>>>>
>>>>> <route path="/pndaconsole/**">
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
>>>>> to="request.url"/>
>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>> to="response.body"/>
>>>>> </route>
>>>>>
>>>>> My inbound rewrites, now, look like this -
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>> </rule>
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>> </rule>
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>> </rule>
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>>> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/>
>>>>> </rule>
>>>>>
>>>>> The ws://...transport=websocket requests are rewritten as /, so I
>>>>> tried adding an entry in my topology for WEBSOCKET and an additional
>>>>> inbound rule like this -
>>>>>
>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>">
>>>>> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>> </rule>
>>>>>
>>>>> Where the topology entry is the same as PNDACONSOLE except it uses the
>>>>> ws:// protocol specifier. This didn't work at all, and in fact at that
>>>>> point, nothing got through and everything fell through to the default
>>>>> rewrite rule.
>>>>>
>>>>> I've attached the gateway log with debug turned on everywhere
>>>>> (tarballed as it's huge).
>>>>>
>>>>> I guess what I'm missing is a clear idea of what I'm supposed to do in
>>>>> the service/rewrite rules to deal with the websockets request which will
>>>>> arrive with a different protocol specifier and a different query parameter.
>>>>> Should I explicitly map those through to the back end, or is this something
>>>>> Knox handles internally?
>>>>>
>>>>> Any insight into this appreciated, it seems very close to working, and
>>>>> in fact socket.io does it's usual fallback so it is functional after
>>>>> a fashion but it seems like this should fully work given what I see with
>>>>> Zeppelin.
>>>>>
>>>>>
>>>>> Cheers,
>>>>> /ailuropod4
>>>>>
>>>>>
>>>>> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello ailuropod4
>>>>>>
>>>>>> You should not have to reroute to a different host port, this should
>>>>>> work.
>>>>>>
>>>>>> Looking at the responses, it looks like protocol switching happens at
>>>>>> the Knox end, between browser and Knox (going by the url ws://
>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW)
>>>>>> but it should have
>>>>>> been wss:// wonder why this is ws://.
>>>>>>
>>>>>> Also, can you include seperate "ws" rules for your service, similar
>>>>>> to Zeppelin.
>>>>>> Also, make sure you have, inbound routes in service.xml for e.g.
>>>>>>
>>>>>> <routes>
>>>>>> <route path="/zeppelin/ws">
>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound"
>>>>>> to="request.url"/>
>>>>>> </route>
>>>>>>
>>>>>> <route path="/zeppelin/ws**">
>>>>>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/>
>>>>>> </route>
>>>>>> </routes>
>>>>>>
>>>>>> and then reference it in rewrite.xml.
>>>>>>
>>>>>> Looking at the error, it looks like the issue is not with rewrite
>>>>>> rules but with the connection between Knox and the backend.
>>>>>> We can see that Knox is initiating an upgrade request but then for
>>>>>> some reason the backend service is closing/refusing it.
>>>>>> Can you turn up the logging in Knox to Debug and share it if you can
>>>>>> (this should atleast tell us whether rewrite rules are working).
>>>>>>
>>>>>> Also, do you see any errors at the backend service ? it woud be good
>>>>>> to know what the backend is seeing and reasons for it to refuse the upgrade
>>>>>> and also the error code
>>>>>> Unfortunately the error code is gobbled up somewhere in the process.
>>>>>>
>>>>>> Best,
>>>>>> Sandeep
>>>>>>
>>>>>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> And if it helps, the stack trace as a result of this from Knox
>>>>>>> follows.
>>>>>>>
>>>>>>> Can I match a url containing &transport=websocket in one of the
>>>>>>> query parameters and map it through to a service defined as ws:// in the
>>>>>>> topology? At the moment, as above, everything is sent to a http://
>>>>>>> service, maybe this is the crux of the problem?
>>>>>>>
>>>>>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>>>>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>>>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>>>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>>>>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to connect to
>>>>>>> websocket server: java.io.IOException: Connect failure
>>>>>>> java.io.IOException: Connect failure
>>>>>>> at
>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>>>> at
>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>>>> at
>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>> at org.eclipse.jetty.websocket.se
>>>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>>> at org.eclipse.jetty.io
>>>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>>> at org.eclipse.jetty.io
>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>>>> at
>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>>> at
>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>>> switch protocols
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>> ... 4 more
>>>>>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>>>>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing
>>>>>>> connection)
>>>>>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException:
>>>>>>> Connect failure
>>>>>>> at
>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:106)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>>> at org.eclipse.jetty.websocket.co
>>>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>>>> at org.eclipse.jetty.websocket.se
>>>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>>> at org.eclipse.jetty.io
>>>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>>>> at
>>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>>> at org.eclipse.jetty.io
>>>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>>>> at
>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>>> at
>>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: java.io.IOException: Connect failure
>>>>>>> at
>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>>>> at
>>>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>>>> at
>>>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>>> ... 12 more
>>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>>> switch protocols
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>>>> at org.eclipse.jetty.websocket.cl
>>>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>>>> ... 4 more
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>>>>>
>>>>>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Sandeep,
>>>>>>>>>
>>>>>>>>> Here's what's happening -
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1. Request URL:
>>>>>>>>>
>>>>>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>>>>>> 2. Request Method:
>>>>>>>>> GET
>>>>>>>>> 3. Status Code:
>>>>>>>>> 200 OK
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1. Request URL:
>>>>>>>>> ws://
>>>>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>>>>>> 2. Request Method:
>>>>>>>>> GET
>>>>>>>>> 3. Status Code:
>>>>>>>>> 101 Switching Protocols
>>>>>>>>>
>>>>>>>>> So far, so good. But, the next couple of requests fail, with 400
>>>>>>>>> or 500 range errors, and the gateway log complains about protocol errors. I
>>>>>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>>>>>> big picture across.
>>>>>>>>>
>>>>>>>>> Looking at tcpdump, what's happening is that ws:// request is
>>>>>>>>> being nerfed to a GET /, which on my backend happens to resolve to a nice
>>>>>>>>> front page of HTML. Websockets obviously panics as this isn't what it
>>>>>>>>> expected, and then the whole process repeats. On my backend, /
>>>>>>>>> socket.io is routed to the socket endpoint, whereas / isn't. So,
>>>>>>>>> my theory is that if I can somehow convince Knox to send out the ws://
>>>>>>>>> request with the correct path including /socket.io, it might work.
>>>>>>>>>
>>>>>>>>> At the moment my service/rewrite sections are very simple, I pick
>>>>>>>>> up requests with socket.io and send them off to the backend.
>>>>>>>>>
>>>>>>>>> services.xml -
>>>>>>>>>
>>>>>>>>> <policies>
>>>>>>>>> <policy role="webappsec"/>
>>>>>>>>> <policy role="authentication" name="Anonymous"/>
>>>>>>>>> <policy role="rewrite"/>
>>>>>>>>> <policy role="authorization"/>
>>>>>>>>> </policies>
>>>>>>>>> <routes>
>>>>>>>>> <route path="/pndaconsole">
>>>>>>>>> </route>
>>>>>>>>> <route path="/pndaconsole/**">
>>>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>>>>> to="response.body"/>
>>>>>>>>> </route>
>>>>>>>>> </routes>
>>>>>>>>>
>>>>>>>>> rewrite.xml -
>>>>>>>>>
>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>>>>> </rule>
>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/
>>>>>>>>> socket.io/?{**} <http://socket.io/?%7B**%7D>"/>
>>>>>>>>> </rule>
>>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>>>>> </rule>
>>>>>>>>> (I've snipped the rest of the rules handling outbound rewrites as
>>>>>>>>> I don't believe they're relevant, but let me know if you think otherwise)
>>>>>>>>>
>>>>>>>>> By all accounts it looks like it's *nearly* working, but I'm
>>>>>>>>> stumped on where my /socket.io path is going. I was considering
>>>>>>>>> routing all websockets stuff to a completely separate backend/port so that
>>>>>>>>> it would go to the right place regardless of how it was rewritten, but that
>>>>>>>>> seems a bit extreme and I was hoping to use this approach.
>>>>>>>>>
>>>>>>>>> My end to end is -
>>>>>>>>>
>>>>>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>>>>>> websockets backend in nodejs
>>>>>>>>>
>>>>>>>>> --> flat files, being an angularjs app
>>>>>>>>>
>>>>>>>>> We don't have to use socket.io, I guess, but it is currently used
>>>>>>>>> everywhere so I'd rather avoid swapping that out to get this working if
>>>>>>>>> possible. It's a fairly common library.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <
>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I am not very familiar with socket.io apps, so you might have to
>>>>>>>>>> fill me in about what you are trying to do.
>>>>>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>>>>>
>>>>>>>>>> What version of Knox are you using ?
>>>>>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has
>>>>>>>>>> some examples in the comments.
>>>>>>>>>>
>>>>>>>>>> What is the ws:// url you are trying to rewrite and the rules you
>>>>>>>>>> are using ?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Sandeep
>>>>>>>>>>
>>>>>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I'm struggling to get a simple socket.io based application
>>>>>>>>>>> working through Knox.
>>>>>>>>>>>
>>>>>>>>>>> I see websockets is supported, but the upgrade handshake seems
>>>>>>>>>>> to be nerfed on the way out, specifically the /socket.io/? etc
>>>>>>>>>>> part simply becomes /.
>>>>>>>>>>>
>>>>>>>>>>> I've tried lots of permutations - does anyone have a working
>>>>>>>>>>> example they can point me towards? I've looked at the Zeppelin approach,
>>>>>>>>>>> but this isn't quite the same thing (not socket.io).
>>>>>>>>>>>
>>>>>>>>>>> Any help appreciated!
>>>>>>>>>>>
>>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: socket.io in knox
Posted by T Smith <ai...@gmail.com>.
Hi Sandeep,
I've had a look at this and here's what I think - maybe we should take this
to knox-developers, I'm happy to help work out a solution but I'm going to
need some insight into the design intent around this module.
a) The existing unit test is hard-coded to test a backend URL that ends in
/ws. This corresponds to a special case in the code for Zeppelin, so the
other paths are never tested, and hence the unit test always passes.
b) createWebSocket passes only a path, so the query component could never
be considered by getMatchedBackendURL
c) getMatchedBackendURL doesn't seem to base its logic on the rewrite
rules at all, in the case where the backend URL doesn't end in /ws it
appends the remainder of the path to the backend (you can demonstrate this
by altering the test case to remove the /ws from the backend URL and
running the existing test - the rewrite rule is {**}/channels but the
result is simply /channels).
d) Due to (a) this is a moot point, but the unit test doesn't check that
the path was rewritten as expected, so it will pass regardless.
I think if we address (b) to pass a concatenation of the path and query
then it might work for cases like mine, but this doesn't address (c) which
will affect anyone wanting to control rewrites. I guess addressing (a) and
(d) would help in the longer term.
Hope this makes sense and I haven't completely missed the point :-) It does
explain the behaviour I'm seeing. How would you like to proceed here?
Cheers,
/ailuropod4
On Mon, Jun 18, 2018 at 7:26 PM, Sandeep Moré <mo...@gmail.com> wrote:
> may be you are right about not not properly handeling the query params and
> we might need to fix this regex:
>
> final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
>
> You should be able to do a quick test using a this Unit Test
> <https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
>
> Best,
> Sandeep
>
>
>
>
> On Mon, Jun 18, 2018 at 1:55 PM T Smith <ai...@gmail.com> wrote:
>
>> Sorry, bad example.
>>
>> I send ws://<host>:<port>/gateway/pnda/pndaconsole/socke
>> t.io/?EIO=3&transport=websocket
>>
>> Knox sends /socket.io/
>>
>> That yields a 400 error, Knox throws org.eclipse.jetty.websocket.api.UpgradeException:
>> Didn't switch protocols
>>
>>
>> On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ai...@gmail.com> wrote:
>>
>>> Hi Sandeep,
>>>
>>> Back to fighting with this. Through some nginx debugging on my backend
>>> I've come to the conclusion that Knox isn't sending the query parameters on
>>> to the backend, regardless of what rewrite rules I specify.
>>>
>>> I.e. I send /gateway/pnda/pndaconsole/socket.io/?EIO=3&
>>> transport=polling&t=MDGlYhd
>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>
>>> Knox sends /socket.io/
>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>
>>> This yields a 400 error, and Knox drops the websocket to the backend.
>>> What was confusing me was the 101 in the browser, but I see now that Knox
>>> is terminating the connection with the browser on one side and opening
>>> another connection with the backend on the other.
>>>
>>> I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc
>>> but I'm wondering if this code needs further work to support query
>>> parameters? What do you think?
>>>
>>> Thanks in advance!
>>>
>>> /ailuropod4
>>>
>>> On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com> wrote:
>>>
>>>> Hi Sandeep,
>>>>
>>>> Looking at my nginx server, it never sees the transport=websocket ws://
>>>> protocol request, but it does see a http request for / at the corresponding
>>>> time, so I think perhaps the root problem here is in that rewrite, as you
>>>> mention. By the way, the reason for ws:// and not wss:// is I switched off
>>>> TLS to remove another potential source of issue, so it's all plain http. I
>>>> saw the same behaviour over https.
>>>>
>>>> My services, now, look like this -
>>>>
>>>> <route path="/pndaconsole/socket.io">
>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
>>>> to="request.url"/>
>>>> </route>
>>>>
>>>> <route path="/pndaconsole/metrics">
>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>> to="request.url"/>
>>>> </route>
>>>>
>>>> <route path="/pndaconsole/">
>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
>>>> to="request.url"/>
>>>> </route>
>>>>
>>>> <route path="/pndaconsole/**">
>>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
>>>> to="request.url"/>
>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>> to="response.body"/>
>>>> </route>
>>>>
>>>> My inbound rewrites, now, look like this -
>>>>
>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>> pattern="*://*:*/**/pndaconsole/">
>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>> </rule>
>>>>
>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>> <http://socket.io/?%7B**%7D>">
>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>>>> <http://socket.io/?%7B**%7D>"/>
>>>> </rule>
>>>>
>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>> </rule>
>>>>
>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
>>>> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/>
>>>> </rule>
>>>>
>>>> The ws://...transport=websocket requests are rewritten as /, so I tried
>>>> adding an entry in my topology for WEBSOCKET and an additional inbound rule
>>>> like this -
>>>>
>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>>>> <http://socket.io/?%7B**%7D>">
>>>> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
>>>> <http://socket.io/?%7B**%7D>"/>
>>>> </rule>
>>>>
>>>> Where the topology entry is the same as PNDACONSOLE except it uses the
>>>> ws:// protocol specifier. This didn't work at all, and in fact at that
>>>> point, nothing got through and everything fell through to the default
>>>> rewrite rule.
>>>>
>>>> I've attached the gateway log with debug turned on everywhere
>>>> (tarballed as it's huge).
>>>>
>>>> I guess what I'm missing is a clear idea of what I'm supposed to do in
>>>> the service/rewrite rules to deal with the websockets request which will
>>>> arrive with a different protocol specifier and a different query parameter.
>>>> Should I explicitly map those through to the back end, or is this something
>>>> Knox handles internally?
>>>>
>>>> Any insight into this appreciated, it seems very close to working, and
>>>> in fact socket.io does it's usual fallback so it is functional after a
>>>> fashion but it seems like this should fully work given what I see with
>>>> Zeppelin.
>>>>
>>>>
>>>> Cheers,
>>>> /ailuropod4
>>>>
>>>>
>>>> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <mo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello ailuropod4
>>>>>
>>>>> You should not have to reroute to a different host port, this should
>>>>> work.
>>>>>
>>>>> Looking at the responses, it looks like protocol switching happens at
>>>>> the Knox end, between browser and Knox (going by the url ws://
>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/
>>>>> socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW) but it
>>>>> should have
>>>>> been wss:// wonder why this is ws://.
>>>>>
>>>>> Also, can you include seperate "ws" rules for your service, similar to
>>>>> Zeppelin.
>>>>> Also, make sure you have, inbound routes in service.xml for e.g.
>>>>>
>>>>> <routes>
>>>>> <route path="/zeppelin/ws">
>>>>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound"
>>>>> to="request.url"/>
>>>>> </route>
>>>>>
>>>>> <route path="/zeppelin/ws**">
>>>>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/>
>>>>> </route>
>>>>> </routes>
>>>>>
>>>>> and then reference it in rewrite.xml.
>>>>>
>>>>> Looking at the error, it looks like the issue is not with rewrite
>>>>> rules but with the connection between Knox and the backend.
>>>>> We can see that Knox is initiating an upgrade request but then for
>>>>> some reason the backend service is closing/refusing it.
>>>>> Can you turn up the logging in Knox to Debug and share it if you can
>>>>> (this should atleast tell us whether rewrite rules are working).
>>>>>
>>>>> Also, do you see any errors at the backend service ? it woud be good
>>>>> to know what the backend is seeing and reasons for it to refuse the upgrade
>>>>> and also the error code
>>>>> Unfortunately the error code is gobbled up somewhere in the process.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com> wrote:
>>>>>
>>>>>> And if it helps, the stack trace as a result of this from Knox
>>>>>> follows.
>>>>>>
>>>>>> Can I match a url containing &transport=websocket in one of the query
>>>>>> parameters and map it through to a service defined as ws:// in the
>>>>>> topology? At the moment, as above, everything is sent to a http://
>>>>>> service, maybe this is the crux of the problem?
>>>>>>
>>>>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>>>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch
>>>>>> protocols
>>>>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>>>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to
>>>>>> connect to websocket server: java.io.IOException: Connect failure
>>>>>> java.io.IOException: Connect failure
>>>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.
>>>>>> connect(ClientContainer.java:157)
>>>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.
>>>>>> connectToServer(ClientContainer.java:180)
>>>>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.
>>>>>> onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>> at org.eclipse.jetty.websocket.common.events.
>>>>>> JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>> at org.eclipse.jetty.websocket.common.events.
>>>>>> AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(
>>>>>> WebSocketSession.java:421)
>>>>>> at org.eclipse.jetty.websocket.server.
>>>>>> WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(
>>>>>> AbstractEndPoint.java:185)
>>>>>> at org.eclipse.jetty.server.HttpConnection.completed(
>>>>>> HttpConnection.java:345)
>>>>>> at org.eclipse.jetty.server.HttpChannel.handle(
>>>>>> HttpChannel.java:436)
>>>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(
>>>>>> HttpConnection.java:257)
>>>>>> at org.eclipse.jetty.io.AbstractConnection$2.run(
>>>>>> AbstractConnection.java:544)
>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>>>>> QueuedThreadPool.java:635)
>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>>>>>> QueuedThreadPool.java:555)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>> switch protocols
>>>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.
>>>>>> validateResponse(UpgradeConnection.java:314)
>>>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.
>>>>>> read(UpgradeConnection.java:241)
>>>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.
>>>>>> onFillable(UpgradeConnection.java:163)
>>>>>> ... 4 more
>>>>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>>>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing
>>>>>> connection)
>>>>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException:
>>>>>> Connect failure
>>>>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.
>>>>>> onWebSocketConnect(ProxyWebSocketAdapter.java:106)
>>>>>> at org.eclipse.jetty.websocket.common.events.
>>>>>> JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>>> at org.eclipse.jetty.websocket.common.events.
>>>>>> AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(
>>>>>> WebSocketSession.java:421)
>>>>>> at org.eclipse.jetty.websocket.server.
>>>>>> WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(
>>>>>> AbstractEndPoint.java:185)
>>>>>> at org.eclipse.jetty.server.HttpConnection.completed(
>>>>>> HttpConnection.java:345)
>>>>>> at org.eclipse.jetty.server.HttpChannel.handle(
>>>>>> HttpChannel.java:436)
>>>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(
>>>>>> HttpConnection.java:257)
>>>>>> at org.eclipse.jetty.io.AbstractConnection$2.run(
>>>>>> AbstractConnection.java:544)
>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>>>>> QueuedThreadPool.java:635)
>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>>>>>> QueuedThreadPool.java:555)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by: java.io.IOException: Connect failure
>>>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.
>>>>>> connect(ClientContainer.java:157)
>>>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.
>>>>>> connectToServer(ClientContainer.java:180)
>>>>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.
>>>>>> onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>>> ... 12 more
>>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>>> switch protocols
>>>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.
>>>>>> validateResponse(UpgradeConnection.java:314)
>>>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.
>>>>>> read(UpgradeConnection.java:241)
>>>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.
>>>>>> onFillable(UpgradeConnection.java:163)
>>>>>> ... 4 more
>>>>>>
>>>>>>
>>>>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>>>>
>>>>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Sandeep,
>>>>>>>>
>>>>>>>> Here's what's happening -
>>>>>>>>
>>>>>>>>
>>>>>>>> 1. Request URL:
>>>>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/
>>>>>>>> socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>>>> 2. Request Method:
>>>>>>>> GET
>>>>>>>> 3. Status Code:
>>>>>>>> 200 OK
>>>>>>>>
>>>>>>>>
>>>>>>>> 1. Request URL:
>>>>>>>> ws://34.244.121.78:8443/gateway/pnda/pndaconsole/
>>>>>>>> socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW>
>>>>>>>> 2. Request Method:
>>>>>>>> GET
>>>>>>>> 3. Status Code:
>>>>>>>> 101 Switching Protocols
>>>>>>>>
>>>>>>>> So far, so good. But, the next couple of requests fail, with 400 or
>>>>>>>> 500 range errors, and the gateway log complains about protocol errors. I
>>>>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>>>>> big picture across.
>>>>>>>>
>>>>>>>> Looking at tcpdump, what's happening is that ws:// request is being
>>>>>>>> nerfed to a GET /, which on my backend happens to resolve to a nice front
>>>>>>>> page of HTML. Websockets obviously panics as this isn't what it expected,
>>>>>>>> and then the whole process repeats. On my backend, /socket.io is
>>>>>>>> routed to the socket endpoint, whereas / isn't. So, my theory is that if I
>>>>>>>> can somehow convince Knox to send out the ws:// request with the correct
>>>>>>>> path including /socket.io, it might work.
>>>>>>>>
>>>>>>>> At the moment my service/rewrite sections are very simple, I pick
>>>>>>>> up requests with socket.io and send them off to the backend.
>>>>>>>>
>>>>>>>> services.xml -
>>>>>>>>
>>>>>>>> <policies>
>>>>>>>> <policy role="webappsec"/>
>>>>>>>> <policy role="authentication" name="Anonymous"/>
>>>>>>>> <policy role="rewrite"/>
>>>>>>>> <policy role="authorization"/>
>>>>>>>> </policies>
>>>>>>>> <routes>
>>>>>>>> <route path="/pndaconsole">
>>>>>>>> </route>
>>>>>>>> <route path="/pndaconsole/**">
>>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>>>> to="response.body"/>
>>>>>>>> </route>
>>>>>>>> </routes>
>>>>>>>>
>>>>>>>> rewrite.xml -
>>>>>>>>
>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>>>> </rule>
>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/
>>>>>>>> socket.io/?{**} <http://socket.io/?%7B**%7D>"/>
>>>>>>>> </rule>
>>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>>>> </rule>
>>>>>>>> (I've snipped the rest of the rules handling outbound rewrites as I
>>>>>>>> don't believe they're relevant, but let me know if you think otherwise)
>>>>>>>>
>>>>>>>> By all accounts it looks like it's *nearly* working, but I'm
>>>>>>>> stumped on where my /socket.io path is going. I was considering
>>>>>>>> routing all websockets stuff to a completely separate backend/port so that
>>>>>>>> it would go to the right place regardless of how it was rewritten, but that
>>>>>>>> seems a bit extreme and I was hoping to use this approach.
>>>>>>>>
>>>>>>>> My end to end is -
>>>>>>>>
>>>>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>>>>> websockets backend in nodejs
>>>>>>>>
>>>>>>>> --> flat files, being an angularjs app
>>>>>>>>
>>>>>>>> We don't have to use socket.io, I guess, but it is currently used
>>>>>>>> everywhere so I'd rather avoid swapping that out to get this working if
>>>>>>>> possible. It's a fairly common library.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <
>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I am not very familiar with socket.io apps, so you might have to
>>>>>>>>> fill me in about what you are trying to do.
>>>>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>>>>
>>>>>>>>> What version of Knox are you using ?
>>>>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has
>>>>>>>>> some examples in the comments.
>>>>>>>>>
>>>>>>>>> What is the ws:// url you are trying to rewrite and the rules you
>>>>>>>>> are using ?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Sandeep
>>>>>>>>>
>>>>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I'm struggling to get a simple socket.io based application
>>>>>>>>>> working through Knox.
>>>>>>>>>>
>>>>>>>>>> I see websockets is supported, but the upgrade handshake seems to
>>>>>>>>>> be nerfed on the way out, specifically the /socket.io/? etc part
>>>>>>>>>> simply becomes /.
>>>>>>>>>>
>>>>>>>>>> I've tried lots of permutations - does anyone have a working
>>>>>>>>>> example they can point me towards? I've looked at the Zeppelin approach,
>>>>>>>>>> but this isn't quite the same thing (not socket.io).
>>>>>>>>>>
>>>>>>>>>> Any help appreciated!
>>>>>>>>>>
>>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: socket.io in knox
Posted by Sandeep Moré <mo...@gmail.com>.
may be you are right about not not properly handeling the query params and
we might need to fix this regex:
final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
You should be able to do a quick test using a this Unit Test
<https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
Best,
Sandeep
On Mon, Jun 18, 2018 at 1:55 PM T Smith <ai...@gmail.com> wrote:
> Sorry, bad example.
>
> I send ws://<host>:<port>/gateway/pnda/pndaconsole/
> socket.io/?EIO=3&transport=websocket
>
> Knox sends /socket.io/
>
> That yields a 400 error, Knox throws
> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>
>
> On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ai...@gmail.com> wrote:
>
>> Hi Sandeep,
>>
>> Back to fighting with this. Through some nginx debugging on my backend
>> I've come to the conclusion that Knox isn't sending the query parameters on
>> to the backend, regardless of what rewrite rules I specify.
>>
>> I.e. I send
>> /gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>
>> Knox sends /socket.io/
>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>
>> This yields a 400 error, and Knox drops the websocket to the backend.
>> What was confusing me was the 101 in the browser, but I see now that Knox
>> is terminating the connection with the browser on one side and opening
>> another connection with the backend on the other.
>>
>> I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc
>> but I'm wondering if this code needs further work to support query
>> parameters? What do you think?
>>
>> Thanks in advance!
>>
>> /ailuropod4
>>
>> On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com> wrote:
>>
>>> Hi Sandeep,
>>>
>>> Looking at my nginx server, it never sees the transport=websocket ws://
>>> protocol request, but it does see a http request for / at the corresponding
>>> time, so I think perhaps the root problem here is in that rewrite, as you
>>> mention. By the way, the reason for ws:// and not wss:// is I switched off
>>> TLS to remove another potential source of issue, so it's all plain http. I
>>> saw the same behaviour over https.
>>>
>>> My services, now, look like this -
>>>
>>> <route path="/pndaconsole/socket.io">
>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
>>> to="request.url"/>
>>> </route>
>>>
>>> <route path="/pndaconsole/metrics">
>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
>>> to="request.url"/>
>>> </route>
>>>
>>> <route path="/pndaconsole/">
>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
>>> to="request.url"/>
>>> </route>
>>>
>>> <route path="/pndaconsole/**">
>>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
>>> to="request.url"/>
>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>> to="response.body"/>
>>> </route>
>>>
>>> My inbound rewrites, now, look like this -
>>>
>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>> pattern="*://*:*/**/pndaconsole/">
>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>> </rule>
>>>
>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>> <http://socket.io/?%7B**%7D>">
>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>>> <http://socket.io/?%7B**%7D>"/>
>>> </rule>
>>>
>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>> pattern="*://*:*/**/pndaconsole/{**}">
>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>> </rule>
>>>
>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
>>> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/>
>>> </rule>
>>>
>>> The ws://...transport=websocket requests are rewritten as /, so I tried
>>> adding an entry in my topology for WEBSOCKET and an additional inbound rule
>>> like this -
>>>
>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>>> <http://socket.io/?%7B**%7D>">
>>> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
>>> <http://socket.io/?%7B**%7D>"/>
>>> </rule>
>>>
>>> Where the topology entry is the same as PNDACONSOLE except it uses the
>>> ws:// protocol specifier. This didn't work at all, and in fact at that
>>> point, nothing got through and everything fell through to the default
>>> rewrite rule.
>>>
>>> I've attached the gateway log with debug turned on everywhere (tarballed
>>> as it's huge).
>>>
>>> I guess what I'm missing is a clear idea of what I'm supposed to do in
>>> the service/rewrite rules to deal with the websockets request which will
>>> arrive with a different protocol specifier and a different query parameter.
>>> Should I explicitly map those through to the back end, or is this something
>>> Knox handles internally?
>>>
>>> Any insight into this appreciated, it seems very close to working, and
>>> in fact socket.io does it's usual fallback so it is functional after a
>>> fashion but it seems like this should fully work given what I see with
>>> Zeppelin.
>>>
>>>
>>> Cheers,
>>> /ailuropod4
>>>
>>>
>>> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <mo...@gmail.com>
>>> wrote:
>>>
>>>> Hello ailuropod4
>>>>
>>>> You should not have to reroute to a different host port, this should
>>>> work.
>>>>
>>>> Looking at the responses, it looks like protocol switching happens at
>>>> the Knox end, between browser and Knox (going by the url ws://
>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW)
>>>> but it should have
>>>> been wss:// wonder why this is ws://.
>>>>
>>>> Also, can you include seperate "ws" rules for your service, similar to
>>>> Zeppelin.
>>>> Also, make sure you have, inbound routes in service.xml for e.g.
>>>>
>>>> <routes>
>>>> <route path="/zeppelin/ws">
>>>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound" to="request.url"/>
>>>> </route>
>>>>
>>>> <route path="/zeppelin/ws**">
>>>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/>
>>>> </route>
>>>> </routes>
>>>>
>>>> and then reference it in rewrite.xml.
>>>>
>>>> Looking at the error, it looks like the issue is not with rewrite rules
>>>> but with the connection between Knox and the backend.
>>>> We can see that Knox is initiating an upgrade request but then for some
>>>> reason the backend service is closing/refusing it.
>>>> Can you turn up the logging in Knox to Debug and share it if you can
>>>> (this should atleast tell us whether rewrite rules are working).
>>>>
>>>> Also, do you see any errors at the backend service ? it woud be good to
>>>> know what the backend is seeing and reasons for it to refuse the upgrade
>>>> and also the error code
>>>> Unfortunately the error code is gobbled up somewhere in the process.
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com> wrote:
>>>>
>>>>> And if it helps, the stack trace as a result of this from Knox follows.
>>>>>
>>>>> Can I match a url containing &transport=websocket in one of the query
>>>>> parameters and map it through to a service defined as ws:// in the
>>>>> topology? At the moment, as above, everything is sent to a http://
>>>>> service, maybe this is the crux of the problem?
>>>>>
>>>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to connect to
>>>>> websocket server: java.io.IOException: Connect failure
>>>>> java.io.IOException: Connect failure
>>>>> at
>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>> at
>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>> at
>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>> at org.eclipse.jetty.websocket.co
>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>> at org.eclipse.jetty.websocket.co
>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>> at org.eclipse.jetty.websocket.co
>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>> at org.eclipse.jetty.websocket.se
>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>> at org.eclipse.jetty.io
>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>> at
>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>> at
>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>> at
>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>> at org.eclipse.jetty.io
>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>> switch protocols
>>>>> at org.eclipse.jetty.websocket.cl
>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>> at org.eclipse.jetty.websocket.cl
>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>> at org.eclipse.jetty.websocket.cl
>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>> ... 4 more
>>>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing
>>>>> connection)
>>>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException: Connect
>>>>> failure
>>>>> at
>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:106)
>>>>> at org.eclipse.jetty.websocket.co
>>>>> mmon.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
>>>>> at org.eclipse.jetty.websocket.co
>>>>> mmon.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
>>>>> at org.eclipse.jetty.websocket.co
>>>>> mmon.WebSocketSession.open(WebSocketSession.java:421)
>>>>> at org.eclipse.jetty.websocket.se
>>>>> rver.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
>>>>> at org.eclipse.jetty.io
>>>>> .AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
>>>>> at
>>>>> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
>>>>> at
>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
>>>>> at
>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>> at org.eclipse.jetty.io
>>>>> .AbstractConnection$2.run(AbstractConnection.java:544)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: java.io.IOException: Connect failure
>>>>> at
>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:157)
>>>>> at
>>>>> org.eclipse.jetty.websocket.jsr356.ClientContainer.connectToServer(ClientContainer.java:180)
>>>>> at
>>>>> org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>>> ... 12 more
>>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>>> switch protocols
>>>>> at org.eclipse.jetty.websocket.cl
>>>>> ient.io.UpgradeConnection.validateResponse(UpgradeConnection.java:314)
>>>>> at org.eclipse.jetty.websocket.cl
>>>>> ient.io.UpgradeConnection.read(UpgradeConnection.java:241)
>>>>> at org.eclipse.jetty.websocket.cl
>>>>> ient.io.UpgradeConnection.onFillable(UpgradeConnection.java:163)
>>>>> ... 4 more
>>>>>
>>>>>
>>>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com> wrote:
>>>>>
>>>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>>>
>>>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Sandeep,
>>>>>>>
>>>>>>> Here's what's happening -
>>>>>>>
>>>>>>>
>>>>>>> 1. Request URL:
>>>>>>>
>>>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd
>>>>>>> 2. Request Method:
>>>>>>> GET
>>>>>>> 3. Status Code:
>>>>>>> 200 OK
>>>>>>>
>>>>>>>
>>>>>>> 1. Request URL:
>>>>>>> ws://
>>>>>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>>>> 2. Request Method:
>>>>>>> GET
>>>>>>> 3. Status Code:
>>>>>>> 101 Switching Protocols
>>>>>>>
>>>>>>> So far, so good. But, the next couple of requests fail, with 400 or
>>>>>>> 500 range errors, and the gateway log complains about protocol errors. I
>>>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>>>> big picture across.
>>>>>>>
>>>>>>> Looking at tcpdump, what's happening is that ws:// request is being
>>>>>>> nerfed to a GET /, which on my backend happens to resolve to a nice front
>>>>>>> page of HTML. Websockets obviously panics as this isn't what it expected,
>>>>>>> and then the whole process repeats. On my backend, /socket.io is
>>>>>>> routed to the socket endpoint, whereas / isn't. So, my theory is that if I
>>>>>>> can somehow convince Knox to send out the ws:// request with the correct
>>>>>>> path including /socket.io, it might work.
>>>>>>>
>>>>>>> At the moment my service/rewrite sections are very simple, I pick up
>>>>>>> requests with socket.io and send them off to the backend.
>>>>>>>
>>>>>>> services.xml -
>>>>>>>
>>>>>>> <policies>
>>>>>>> <policy role="webappsec"/>
>>>>>>> <policy role="authentication" name="Anonymous"/>
>>>>>>> <policy role="rewrite"/>
>>>>>>> <policy role="authorization"/>
>>>>>>> </policies>
>>>>>>> <routes>
>>>>>>> <route path="/pndaconsole">
>>>>>>> </route>
>>>>>>> <route path="/pndaconsole/**">
>>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>>> to="response.body"/>
>>>>>>> </route>
>>>>>>> </routes>
>>>>>>>
>>>>>>> rewrite.xml -
>>>>>>>
>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>>> </rule>
>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/
>>>>>>> socket.io/?{**} <http://socket.io/?%7B**%7D>"/>
>>>>>>> </rule>
>>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>>> </rule>
>>>>>>> (I've snipped the rest of the rules handling outbound rewrites as I
>>>>>>> don't believe they're relevant, but let me know if you think otherwise)
>>>>>>>
>>>>>>> By all accounts it looks like it's *nearly* working, but I'm stumped
>>>>>>> on where my /socket.io path is going. I was considering routing all
>>>>>>> websockets stuff to a completely separate backend/port so that it would go
>>>>>>> to the right place regardless of how it was rewritten, but that seems a bit
>>>>>>> extreme and I was hoping to use this approach.
>>>>>>>
>>>>>>> My end to end is -
>>>>>>>
>>>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>>>> websockets backend in nodejs
>>>>>>>
>>>>>>> --> flat files, being an angularjs app
>>>>>>>
>>>>>>> We don't have to use socket.io, I guess, but it is currently used
>>>>>>> everywhere so I'd rather avoid swapping that out to get this working if
>>>>>>> possible. It's a fairly common library.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <moresandeep@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I am not very familiar with socket.io apps, so you might have to
>>>>>>>> fill me in about what you are trying to do.
>>>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>>>
>>>>>>>> What version of Knox are you using ?
>>>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has some
>>>>>>>> examples in the comments.
>>>>>>>>
>>>>>>>> What is the ws:// url you are trying to rewrite and the rules you
>>>>>>>> are using ?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Sandeep
>>>>>>>>
>>>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I'm struggling to get a simple socket.io based application
>>>>>>>>> working through Knox.
>>>>>>>>>
>>>>>>>>> I see websockets is supported, but the upgrade handshake seems to
>>>>>>>>> be nerfed on the way out, specifically the /socket.io/? etc part
>>>>>>>>> simply becomes /.
>>>>>>>>>
>>>>>>>>> I've tried lots of permutations - does anyone have a working
>>>>>>>>> example they can point me towards? I've looked at the Zeppelin approach,
>>>>>>>>> but this isn't quite the same thing (not socket.io).
>>>>>>>>>
>>>>>>>>> Any help appreciated!
>>>>>>>>>
>>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: socket.io in knox
Posted by T Smith <ai...@gmail.com>.
Sorry, bad example.
I send ws://<host>:<port>/gateway/pnda/pndaconsole/
socket.io/?EIO=3&transport=websocket
Knox sends /socket.io/
That yields a 400 error, Knox throws
org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
On Mon, Jun 18, 2018 at 6:49 PM, T Smith <ai...@gmail.com> wrote:
> Hi Sandeep,
>
> Back to fighting with this. Through some nginx debugging on my backend
> I've come to the conclusion that Knox isn't sending the query parameters on
> to the backend, regardless of what rewrite rules I specify.
>
> I.e. I send /gateway/pnda/pndaconsole/socket.io/?EIO=3&
> transport=polling&t=MDGlYhd
> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>
> Knox sends /socket.io/
> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>
> This yields a 400 error, and Knox drops the websocket to the backend. What
> was confusing me was the 101 in the browser, but I see now that Knox is
> terminating the connection with the browser on one side and opening another
> connection with the backend on the other.
>
> I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc
> but I'm wondering if this code needs further work to support query
> parameters? What do you think?
>
> Thanks in advance!
>
> /ailuropod4
>
> On Thu, May 17, 2018 at 8:57 AM, T Smith <ai...@gmail.com> wrote:
>
>> Hi Sandeep,
>>
>> Looking at my nginx server, it never sees the transport=websocket ws://
>> protocol request, but it does see a http request for / at the corresponding
>> time, so I think perhaps the root problem here is in that rewrite, as you
>> mention. By the way, the reason for ws:// and not wss:// is I switched off
>> TLS to remove another potential source of issue, so it's all plain http. I
>> saw the same behaviour over https.
>>
>> My services, now, look like this -
>>
>> <route path="/pndaconsole/socket.io">
>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket"
>> to="request.url"/>
>> </route>
>>
>> <route path="/pndaconsole/metrics">
>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics"
>> to="request.url"/>
>> </route>
>>
>> <route path="/pndaconsole/">
>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root"
>> to="request.url"/>
>> </route>
>>
>> <route path="/pndaconsole/**">
>> <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path"
>> to="request.url"/>
>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>> to="response.body"/>
>> </route>
>>
>> My inbound rewrites, now, look like this -
>>
>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>> pattern="*://*:*/**/pndaconsole/">
>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>> </rule>
>>
>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>> <http://socket.io/?%7B**%7D>">
>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>> <http://socket.io/?%7B**%7D>"/>
>> </rule>
>>
>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>> pattern="*://*:*/**/pndaconsole/{**}">
>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>> </rule>
>>
>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics"
>> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/>
>> </rule>
>>
>> The ws://...transport=websocket requests are rewritten as /, so I tried
>> adding an entry in my topology for WEBSOCKET and an additional inbound rule
>> like this -
>>
>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>> <http://socket.io/?%7B**%7D>">
>> <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**}
>> <http://socket.io/?%7B**%7D>"/>
>> </rule>
>>
>> Where the topology entry is the same as PNDACONSOLE except it uses the
>> ws:// protocol specifier. This didn't work at all, and in fact at that
>> point, nothing got through and everything fell through to the default
>> rewrite rule.
>>
>> I've attached the gateway log with debug turned on everywhere (tarballed
>> as it's huge).
>>
>> I guess what I'm missing is a clear idea of what I'm supposed to do in
>> the service/rewrite rules to deal with the websockets request which will
>> arrive with a different protocol specifier and a different query parameter.
>> Should I explicitly map those through to the back end, or is this something
>> Knox handles internally?
>>
>> Any insight into this appreciated, it seems very close to working, and in
>> fact socket.io does it's usual fallback so it is functional after a
>> fashion but it seems like this should fully work given what I see with
>> Zeppelin.
>>
>>
>> Cheers,
>> /ailuropod4
>>
>>
>> On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> Hello ailuropod4
>>>
>>> You should not have to reroute to a different host port, this should
>>> work.
>>>
>>> Looking at the responses, it looks like protocol switching happens at
>>> the Knox end, between browser and Knox (going by the url ws://
>>> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?
>>> EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW) but it should have
>>> been wss:// wonder why this is ws://.
>>>
>>> Also, can you include seperate "ws" rules for your service, similar to
>>> Zeppelin.
>>> Also, make sure you have, inbound routes in service.xml for e.g.
>>>
>>> <routes>
>>> <route path="/zeppelin/ws">
>>> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound" to="request.url"/>
>>> </route>
>>>
>>> <route path="/zeppelin/ws**">
>>> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/>
>>> </route>
>>> </routes>
>>>
>>> and then reference it in rewrite.xml.
>>>
>>> Looking at the error, it looks like the issue is not with rewrite rules
>>> but with the connection between Knox and the backend.
>>> We can see that Knox is initiating an upgrade request but then for some
>>> reason the backend service is closing/refusing it.
>>> Can you turn up the logging in Knox to Debug and share it if you can
>>> (this should atleast tell us whether rewrite rules are working).
>>>
>>> Also, do you see any errors at the backend service ? it woud be good to
>>> know what the backend is seeing and reasons for it to refuse the upgrade
>>> and also the error code
>>> Unfortunately the error code is gobbled up somewhere in the process.
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Sat, May 12, 2018 at 3:16 AM, T Smith <ai...@gmail.com> wrote:
>>>
>>>> And if it helps, the stack trace as a result of this from Knox follows.
>>>>
>>>> Can I match a url containing &transport=websocket in one of the query
>>>> parameters and map it through to a service defined as ws:// in the
>>>> topology? At the moment, as above, everything is sent to a http://
>>>> service, maybe this is the crux of the problem?
>>>>
>>>> 2018-05-12 07:09:47,361 ERROR gateway.websockets
>>>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error:
>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch
>>>> protocols
>>>> 2018-05-12 07:09:47,362 ERROR gateway.websockets
>>>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to
>>>> connect to websocket server: java.io.IOException: Connect failure
>>>> java.io.IOException: Connect failure
>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(C
>>>> lientContainer.java:157)
>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connectTo
>>>> Server(ClientContainer.java:180)
>>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW
>>>> ebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent
>>>> Driver.onConnect(JettyListenerEventDriver.java:87)
>>>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive
>>>> r.openSession(AbstractEventDriver.java:227)
>>>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(Web
>>>> SocketSession.java:421)
>>>> at org.eclipse.jetty.websocket.server.WebSocketServerConnection
>>>> .onOpen(WebSocketServerConnection.java:72)
>>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoi
>>>> nt.java:185)
>>>> at org.eclipse.jetty.server.HttpConnection.completed(HttpConnec
>>>> tion.java:345)
>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java
>>>> :436)
>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>>>> ction.java:257)
>>>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec
>>>> tion.java:544)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>>>> ThreadPool.java:635)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT
>>>> hreadPool.java:555)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>> switch protocols
>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.vali
>>>> dateResponse(UpgradeConnection.java:314)
>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.read
>>>> (UpgradeConnection.java:241)
>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.onFi
>>>> llable(UpgradeConnection.java:163)
>>>> ... 4 more
>>>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter
>>>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing
>>>> connection)
>>>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException: Connect
>>>> failure
>>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW
>>>> ebSocketConnect(ProxyWebSocketAdapter.java:106)
>>>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent
>>>> Driver.onConnect(JettyListenerEventDriver.java:87)
>>>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive
>>>> r.openSession(AbstractEventDriver.java:227)
>>>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(Web
>>>> SocketSession.java:421)
>>>> at org.eclipse.jetty.websocket.server.WebSocketServerConnection
>>>> .onOpen(WebSocketServerConnection.java:72)
>>>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoi
>>>> nt.java:185)
>>>> at org.eclipse.jetty.server.HttpConnection.completed(HttpConnec
>>>> tion.java:345)
>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java
>>>> :436)
>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>>>> ction.java:257)
>>>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec
>>>> tion.java:544)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>>>> ThreadPool.java:635)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT
>>>> hreadPool.java:555)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: java.io.IOException: Connect failure
>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(C
>>>> lientContainer.java:157)
>>>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connectTo
>>>> Server(ClientContainer.java:180)
>>>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW
>>>> ebSocketConnect(ProxyWebSocketAdapter.java:97)
>>>> ... 12 more
>>>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't
>>>> switch protocols
>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.vali
>>>> dateResponse(UpgradeConnection.java:314)
>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.read
>>>> (UpgradeConnection.java:241)
>>>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.onFi
>>>> llable(UpgradeConnection.java:163)
>>>> ... 4 more
>>>>
>>>>
>>>> On Fri, May 11, 2018 at 9:04 PM, T Smith <ai...@gmail.com> wrote:
>>>>
>>>>> Sorry, the other question was version. I'm using 1.0.0.
>>>>>
>>>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <ai...@gmail.com> wrote:
>>>>>
>>>>>> Hi Sandeep,
>>>>>>
>>>>>> Here's what's happening -
>>>>>>
>>>>>>
>>>>>> 1. Request URL:
>>>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io
>>>>>> /?EIO=3&transport=polling&t=MDGlYhd
>>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd>
>>>>>> 2. Request Method:
>>>>>> GET
>>>>>> 3. Status Code:
>>>>>> 200 OK
>>>>>>
>>>>>>
>>>>>> 1. Request URL:
>>>>>> ws://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?
>>>>>> EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW
>>>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW>
>>>>>> 2. Request Method:
>>>>>> GET
>>>>>> 3. Status Code:
>>>>>> 101 Switching Protocols
>>>>>>
>>>>>> So far, so good. But, the next couple of requests fail, with 400 or
>>>>>> 500 range errors, and the gateway log complains about protocol errors. I
>>>>>> can supply these if you need them, I'm trying to be concise here to get the
>>>>>> big picture across.
>>>>>>
>>>>>> Looking at tcpdump, what's happening is that ws:// request is being
>>>>>> nerfed to a GET /, which on my backend happens to resolve to a nice front
>>>>>> page of HTML. Websockets obviously panics as this isn't what it expected,
>>>>>> and then the whole process repeats. On my backend, /socket.io is
>>>>>> routed to the socket endpoint, whereas / isn't. So, my theory is that if I
>>>>>> can somehow convince Knox to send out the ws:// request with the correct
>>>>>> path including /socket.io, it might work.
>>>>>>
>>>>>> At the moment my service/rewrite sections are very simple, I pick up
>>>>>> requests with socket.io and send them off to the backend.
>>>>>>
>>>>>> services.xml -
>>>>>>
>>>>>> <policies>
>>>>>> <policy role="webappsec"/>
>>>>>> <policy role="authentication" name="Anonymous"/>
>>>>>> <policy role="rewrite"/>
>>>>>> <policy role="authorization"/>
>>>>>> </policies>
>>>>>> <routes>
>>>>>> <route path="/pndaconsole">
>>>>>> </route>
>>>>>> <route path="/pndaconsole/**">
>>>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app"
>>>>>> to="response.body"/>
>>>>>> </route>
>>>>>> </routes>
>>>>>>
>>>>>> rewrite.xml -
>>>>>>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root"
>>>>>> pattern="*://*:*/**/pndaconsole/">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/>
>>>>>> </rule>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket"
>>>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**}
>>>>>> <http://socket.io/?%7B**%7D>"/>
>>>>>> </rule>
>>>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path"
>>>>>> pattern="*://*:*/**/pndaconsole/{**}">
>>>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/>
>>>>>> </rule>
>>>>>> (I've snipped the rest of the rules handling outbound rewrites as I
>>>>>> don't believe they're relevant, but let me know if you think otherwise)
>>>>>>
>>>>>> By all accounts it looks like it's *nearly* working, but I'm stumped
>>>>>> on where my /socket.io path is going. I was considering routing all
>>>>>> websockets stuff to a completely separate backend/port so that it would go
>>>>>> to the right place regardless of how it was rewritten, but that seems a bit
>>>>>> extreme and I was hoping to use this approach.
>>>>>>
>>>>>> My end to end is -
>>>>>>
>>>>>> client, an angularjs app --> knox --> nginx --> reverse proxied
>>>>>> websockets backend in nodejs
>>>>>>
>>>>>> --> flat files, being an angularjs app
>>>>>>
>>>>>> We don't have to use socket.io, I guess, but it is currently used
>>>>>> everywhere so I'd rather avoid swapping that out to get this working if
>>>>>> possible. It's a fairly common library.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <mo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I am not very familiar with socket.io apps, so you might have to
>>>>>>> fill me in about what you are trying to do.
>>>>>>> Looks like you are having issue rewriting Websocket url.
>>>>>>>
>>>>>>> What version of Knox are you using ?
>>>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
>>>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has some
>>>>>>> examples in the comments.
>>>>>>>
>>>>>>> What is the ws:// url you are trying to rewrite and the rules you
>>>>>>> are using ?
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <ai...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm struggling to get a simple socket.io based application working
>>>>>>>> through Knox.
>>>>>>>>
>>>>>>>> I see websockets is supported, but the upgrade handshake seems to
>>>>>>>> be nerfed on the way out, specifically the /socket.io/? etc part
>>>>>>>> simply becomes /.
>>>>>>>>
>>>>>>>> I've tried lots of permutations - does anyone have a working
>>>>>>>> example they can point me towards? I've looked at the Zeppelin approach,
>>>>>>>> but this isn't quite the same thing (not socket.io).
>>>>>>>>
>>>>>>>> Any help appreciated!
>>>>>>>>
>>>>>>>> /ailuropod4 <ai...@gmail.com>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>