POST requests being processed twice by felix HTTP

[Tom Quarendon <to...@worldprogramming.com>]

We've got an off problem that we're trying to diagnose.

The symptoms are that a POST request is sent, and the servlet gets called twice to process it, resulting in a duplicate key violation on the database.

This appears to occur when reconfiguration of components occurs whilst the request is being processed.
So what appears to happen is that the request is received, and the servlet gets called to process it.
Before the response is generated, the servlet component gets recycled, destroyed and recreated. Felix HTTP whiteboard responds to these lifecycle events and presumably reconfigures jetty in response.
Someone, somewhere then decides that the request should be retried, and the same request then gets passed to the new servlet, and the response from that gets sent back to the client.

This primarily happens in testing environments when the world is being brought up and down for each test, but you can actually reproduce it in the debugger by putting a breakpoint in the servlet, then when the breakpoint is hit, edit the code and save it. The servlet gets recycled and the request is processed twice.

In our case the servlet is being registered using declarative services, using the HTTP whiteboard service.

Is this a known problem?
Is there a known solution to this problem?

Where is the most likely culprit? It seems highly unlikely that jetty would be trying a POST request, that would appear to fundamentally break the contract for POST. GET, PUT, DELETE fine, they are all idempotent, but not POST.

I haven't yet delved into the felix HTTP whiteboard code to try and understand its relationship with jetty, so I've no idea whether felix http would get involved, or whether the problem would lie with jetty. Is it likely that the problem is at the felix layer?

Any suggestions for investigation would be most appreciated.

Thanks.

RE: POST requests being processed twice by felix HTTP

[Tom Quarendon <to...@worldprogramming.com>]

Well as I say, at the moment I can't dig into anything as I don't appear to have any way to get matching source code for the felix HTTP bundles, all of the download links appear to be broken.

Since I've realised that actually fixing the double POST doesn't really help us anyway, that's now less of a priority than simply ensuring that the tests "wait until everything is ready" before execution and then "wait until everything is shutdown" after execution. At the moment we are doing the former by simply polling special "ping" endpoints until we get an OK response, but we don't do anything to check that things have shutdown before moving to the next test, which I think is where the main problem lies. 

We will investigate querying the whiteboard service directly, especially on test "cleanup" (i.e junit @After method) to see if that makes things more reliable.

-----Original Message-----
From: Carsten Ziegeler [mailto:cziegeler@apache.org] 
Sent: 21 December 2016 10:58
To: users@felix.apache.org
Subject: Re: POST requests being processed twice by felix HTTP

Hi,

you can use the runtime service defined by the Http Whiteboard spec to find out whether something is still registered. The only downside is that you need to poll and check the DTOs until the desired state is reached. But for testing this should be totally fine.

However, it would be great if we could further pinpoint the problem and avoid the duplicate requests in general.

Carsten

Tom Quarendon wrote
>> If that is not the case setting a break point in the DispatcherServlet from Felix will tell you whether the request is called twice from outside the Felix http implementation (Jetty in that case then) or now.
> 
> The implication of this (I haven't looked into the felix code at this point, and actually all of the source links for the HTTP projects on the felix download page seem to be broken, I've tried numerous mirrors) is that there is a single servlet that you register with jetty, and then you manage distributing the request to the relevant servlet?
> 
> We are as sure as we can be that there is only one HTTP request made, and two calls to the server. We've turned on Jetty tracing and logged our own servlet and this appears to be the case. Up until recently we haven't had a reliable way to reproduce it, but the technique of editing the code in a debug session so that you can force it to redeploy the component while it's in the middle of processing the request appears to work.
> 
> Since my original post, I realise that in some ways it's not interesting why the problem occurs, or actually "solving" it (clearly, to me, POST being processed twice is a bug somewhere).
> Instead what we want is to make our tests reliable. And I think what this means is that we need a way of waiting until everything is shut down after each test. If we had a way of detecting whether all servlets had been deregistered, then that might help. We suspect that although our test has finished, background reconfiguration occurs within OSGi, and this is still happening when the next test starts.  It then makes a request that gets handled by a servlet that is in the process of being shut down. If we could avoid all of that, and ensure that we go back to a clean slate each time, that's probably what we really need.
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
> For additional commands, e-mail: users-help@felix.apache.org
> 
> 


 

--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org

Re: POST requests being processed twice by felix HTTP

[Carsten Ziegeler <cz...@apache.org>]

Hi,

you can use the runtime service defined by the Http Whiteboard spec to
find out whether something is still registered. The only downside is
that you need to poll and check the DTOs until the desired state is
reached. But for testing this should be totally fine.

However, it would be great if we could further pinpoint the problem and
avoid the duplicate requests in general.

Carsten

Tom Quarendon wrote
>> If that is not the case setting a break point in the DispatcherServlet from Felix will tell you whether the request is called twice from outside the Felix http implementation (Jetty in that case then) or now.
> 
> The implication of this (I haven't looked into the felix code at this point, and actually all of the source links for the HTTP projects on the felix download page seem to be broken, I've tried numerous mirrors) is that there is a single servlet that you register with jetty, and then you manage distributing the request to the relevant servlet?
> 
> We are as sure as we can be that there is only one HTTP request made, and two calls to the server. We've turned on Jetty tracing and logged our own servlet and this appears to be the case. Up until recently we haven't had a reliable way to reproduce it, but the technique of editing the code in a debug session so that you can force it to redeploy the component while it's in the middle of processing the request appears to work.
> 
> Since my original post, I realise that in some ways it's not interesting why the problem occurs, or actually "solving" it (clearly, to me, POST being processed twice is a bug somewhere).
> Instead what we want is to make our tests reliable. And I think what this means is that we need a way of waiting until everything is shut down after each test. If we had a way of detecting whether all servlets had been deregistered, then that might help. We suspect that although our test has finished, background reconfiguration occurs within OSGi, and this is still happening when the next test starts.  It then makes a request that gets handled by a servlet that is in the process of being shut down. If we could avoid all of that, and ensure that we go back to a clean slate each time, that's probably what we really need.
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
> For additional commands, e-mail: users-help@felix.apache.org
> 
> 


 

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org

RE: POST requests being processed twice by felix HTTP

[Tom Quarendon <to...@worldprogramming.com>]

> If that is not the case setting a break point in the DispatcherServlet from Felix will tell you whether the request is called twice from outside the Felix http implementation (Jetty in that case then) or now.

The implication of this (I haven't looked into the felix code at this point, and actually all of the source links for the HTTP projects on the felix download page seem to be broken, I've tried numerous mirrors) is that there is a single servlet that you register with jetty, and then you manage distributing the request to the relevant servlet?

We are as sure as we can be that there is only one HTTP request made, and two calls to the server. We've turned on Jetty tracing and logged our own servlet and this appears to be the case. Up until recently we haven't had a reliable way to reproduce it, but the technique of editing the code in a debug session so that you can force it to redeploy the component while it's in the middle of processing the request appears to work.

Since my original post, I realise that in some ways it's not interesting why the problem occurs, or actually "solving" it (clearly, to me, POST being processed twice is a bug somewhere).
Instead what we want is to make our tests reliable. And I think what this means is that we need a way of waiting until everything is shut down after each test. If we had a way of detecting whether all servlets had been deregistered, then that might help. We suspect that although our test has finished, background reconfiguration occurs within OSGi, and this is still happening when the next test starts.  It then makes a request that gets handled by a servlet that is in the process of being shut down. If we could avoid all of that, and ensure that we go back to a clean slate each time, that's probably what we really need.



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org

Re: POST requests being processed twice by felix HTTP

[Carsten Ziegeler <cz...@apache.org>]

Tom Quarendon wrote
> We've got an off problem that we're trying to diagnose.
> 
> The symptoms are that a POST request is sent, and the servlet gets called twice to process it, resulting in a duplicate key violation on the database.
> 
> This appears to occur when reconfiguration of components occurs whilst the request is being processed.
> So what appears to happen is that the request is received, and the servlet gets called to process it.
> Before the response is generated, the servlet component gets recycled, destroyed and recreated. Felix HTTP whiteboard responds to these lifecycle events and presumably reconfigures jetty in response.
> Someone, somewhere then decides that the request should be retried, and the same request then gets passed to the new servlet, and the response from that gets sent back to the client.
> 
> This primarily happens in testing environments when the world is being brought up and down for each test, but you can actually reproduce it in the debugger by putting a breakpoint in the servlet, then when the breakpoint is hit, edit the code and save it. The servlet gets recycled and the request is processed twice.
> 
> In our case the servlet is being registered using declarative services, using the HTTP whiteboard service.
> 
> Is this a known problem?
> Is there a known solution to this problem?
> 
> Where is the most likely culprit? It seems highly unlikely that jetty would be trying a POST request, that would appear to fundamentally break the contract for POST. GET, PUT, DELETE fine, they are all idempotent, but not POST.
> 
> I haven't yet delved into the felix HTTP whiteboard code to try and understand its relationship with jetty, so I've no idea whether felix http would get involved, or whether the problem would lie with jetty. Is it likely that the problem is at the felix layer?
> 
> Any suggestions for investigation would be most appreciated.
> 
Hi,

afaik there is nothing in the Felix http implementation which would
re-execute the request. The registration in Jetty is not changed when
servlets are registered or unregistered through OSGi, so at first look
this would rule out Jetty for me as well :)

Are you sure that your client is not sending a second request if the
first fails? Or do you have an error servlet registered that is called?

If that is not the case setting a break point in the DispatcherServlet
from Felix will tell you whether the request is called twice from
outside the Felix http implementation (Jetty in that case then) or now.

Regards

 Carsten

-- 
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org