You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jmeter.apache.org by Philippe Mouawad <ph...@gmail.com> on 2013/03/25 15:17:55 UTC
Strange Behaviour with JavaImpl & Load Balancer
Hello,
I noticed a strange behaviour when Load Testing an application recently.
There was a Load Balancer in the middle:
- Using Java Implementation I had around 8% of error
- Using HC3 or HC4 Implementations it was 0%
I am sure it comes from Load Balancing as if it pointed on only one Apache
then there was no issue.
Now playing around with some Java System properties, I discovered that
setting *-Dhttp.keepAlive=false* fixed the error issue:
-
http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
Another way to fix this was to uncheck KeepAlive checkbox in samplers.
Now my question is:
- Is this regular ? It seems strange to me that HC31 and HC4 impl do not
have this behaviour
- If yes, we should absolutely document this behaviour no ?
- Could it be a JDK 7 bug ?java version "1.7.0_13"
Note that tested application was not perfect as it had many Content
Encoding behaviour and other non fully standard issues.
Thanks for you notes.
Regards
Philippe
Re: Strange Behaviour with JavaImpl & Load Balancer
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello Sebb, Milamber , Rainer,
What's your thoughts on that ?
Regards
Philippe
On Fri, Mar 29, 2013 at 10:03 PM, Philippe Mouawad <
philippe.mouawad@gmail.com> wrote:
> Hello,
> A little update about this topic, after further analysis it turned out
> there was a bug in Load Balancer
> which didn't handle correctly clients that use connection multiplexing.
>
> Regarding this comment on Java Impl
>
> "The API is best suited to single-threaded usage".
>
> Don't you think we should change this ? it does not seem true and If it's
> true, as JMeter is mainly used for Load Testing (ie multiple threads) it
> seems to me weird, either this is wrong or the sentence is ambiguous, if
> true we should remove it or limit it to functional testing.
> As of my tests I think it works fine in multi threaded usage.
>
> What's your opinion on this ?
>
> Regards
> Philippe
>
>
>
> On Tue, Mar 26, 2013 at 7:42 AM, Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
>> Hello sebb,
>> Thanks for all your ideas.
>> My answers below.
>>
>> Regards
>> Philippe
>>
>> On Mon, Mar 25, 2013 at 9:38 PM, sebb <se...@gmail.com> wrote:
>>
>>> On 25 March 2013 19:38, Philippe Mouawad <ph...@gmail.com>
>>> wrote:
>>> > On Monday, March 25, 2013, sebb wrote:
>>> >
>>> >> On 25 March 2013 17:31, Philippe Mouawad <philippe.mouawad@gmail.com
>>> <javascript:;>>
>>> >> wrote:
>>> >> > Hello,
>>> >> > Thanks sebb, see my notes below.
>>> >> >
>>> >> > On Mon, Mar 25, 2013 at 6:21 PM, sebb <sebbaz@gmail.com<javascript:;>>
>>> >> wrote:
>>> >> >
>>> >> >> On 25 March 2013 14:17, Philippe Mouawad <
>>> philippe.mouawad@gmail.com<javascript:;>
>>> >> >
>>> >> >> wrote:
>>> >> >> > Hello,
>>> >> >> > I noticed a strange behaviour when Load Testing an application
>>> >> recently.
>>> >> >> >
>>> >> >> > There was a Load Balancer in the middle:
>>> >> >> >
>>> >> >> > - Using Java Implementation I had around 8% of error
>>> >> >>
>>> >> >> What kinds of error?
>>> >> >>
>>> >> >> Session is lost, so it means load balancer redirects to other
>>> Apache
>>> >> > ignoring Session Stickiness.
>>> >>
>>> >> What controls the session?
>>> >
>>> >
>>> > A cookie one created by LB for session stickyness and one created by
>>> Apache
>>> > HttpServer
>>> >
>>> >>
>>> >> Does it use cookies?
>>> >
>>> > Yes
>>> >
>>> >> Does it rely on the connection in any way?
>>> >>
>>> >> I don't think so. Hc do not have this issue.
>>> >
>>>
>>> I don't think you can assume that, because JMeter's HC impl. uses
>>> connections very differently.
>>>
>>> Maybe it assumes that requests with the same connection are related.
>>> This cannot happen for HC (at least not in JMeter, because we don't
>>> share connections).
>>>
>>> If you wanted to experiment further, you could use one of the HC
>>> multi-threaded examples to test; but that might be a lot of work.
>>>
>>> >
>>> >> >> > - Using HC3 or HC4 Implementations it was 0%
>>> >> >> >
>>> >> >> >
>>> >> >> > I am sure it comes from Load Balancing as if it pointed on only
>>> one
>>> >> >> Apache
>>> >> >> > then there was no issue.
>>> >> >>
>>> >> >> Note that the Java implementation may reuse connections between
>>> threads.
>>> >> >> That's one reason JMeter started using HC3 and HC4.
>>> >> >>
>>> >> >> > Now playing around with some Java System properties, I
>>> discovered that
>>> >> >> > setting *-Dhttp.keepAlive=false* fixed the error issue:
>>> >> >>
>>> >> >> That suggests it might be the connection reuse.
>>> >> >>
>>> >> > Agree
>>> >> >
>>> >> >>
>>> >> >> > -
>>> >> >> >
>>> >> >>
>>> >>
>>> http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
>>> >> >> >
>>> >> >> > Another way to fix this was to uncheck KeepAlive checkbox in
>>> >> samplers.
>>> >> >>
>>> >> >> Likewise, that should stop cross-thread connection sharing, as it
>>> will
>>> >> >> stop any sharing.
>>> >> >>
>>> >> >> >
>>> >> >> > Now my question is:
>>> >> >> >
>>> >> >> > - Is this regular ? It seems strange to me that HC31 and HC4
>>> impl
>>> >> do
>>> >> >> not
>>> >> >> > have this behaviour
>>> >> >> > - If yes, we should absolutely document this behaviour no ?
>>> >> >>
>>> >> >> I think it is; but it could perhaps be made more obvious.
>>> >> >>
>>> >> > Where is it ? Didn't see any mention of -Dhttp.keepAlive in docs
>>> >>
>>> >> The docs don't mention the property but they do mention why the Java
>>> >> implementation is unsuitable; e.g.:
>>> >>
>>> >> >>
>>> >> The Java HTTP implementation has some limitations:
>>> >> * There is no control over how connections are re-used. When a
>>> >> connection is released by JMeter, it may or may not be re-used by the
>>> >> same thread.
>>> >> <<
>>> >>
>>> >> Well reading this I wouldn't have thought we would face such failure.
>>> >
>>>
>> Regarding this "The API is best suited to single-threaded usage". If it's
>> true as JMeter is mainly used for Load Testing (ie multiple threads) it
>> seems to me weird, either this is wrong or the sentence is ambiguous, if
>> true we should remove it or limit it to functional testing.
>>
>>> >
>>> >> By all means add details of the property if you think it would help.
>>> >>
>>> >> I will.
>>> > I am not a big user of JAva Impl , had to as this same application
>>> failed
>>> > to record with HC some requests, ie when played by Http Server Proxy
>>> they
>>> > didn't return response returned by browser or Java Impl.
>>>
>>> That needs fixing.
>>>
>>> My issue is that it's a contract limited in time and I unfortunately
>> don't have time at all to work on this diagnosis and fix, otherwise I would
>> have :-)
>> I think it comes from charset issues as application is not great at this
>> but couldn't investigate further.
>>
>>
>>> > But I must say it came to me as a bad surprise and would like if
>>> keepalive
>>> > is not handled correctly to kind of avoid these cases by adding it to
>>> > startup script of not allowing keepAlive in this case.
>>>
>>> I don't think it's KeepAlive per se, otherwise HC would suffer.
>>> AFAIK disabling KeepAlive forces the connection to be closed, so
>>> obviously it cannot then be used by a different thread.
>>> That should be possible to test by getting the threads to set a cookie
>>> with the thread ID in it.
>>> Or ideally a unique Cookie named after the thread, as that would
>>> reveal any unintentional cookie sharing as there would be multiple
>>> such cookies (I assume it cannot happen).
>>>
>>>
>>
>>> > But this may also only be related to this application and reveal an
>>> > underlying issue in application.
>>> > That's why I was asking about your experiences with Java Impl.
>>> >
>>> >
>>> >> >>
>>> >> >> > - Could it be a JDK 7 bug ?java version "1.7.0_13"
>>> >> >>
>>> >> >> No, it's by design.
>>> >> >>
>>> >> >> In theory the connnection sharing should be OK (otherwise I assume
>>> >> >> Java would not do it), but maybe there is a subtle bug in the
>>> >> >> application that relies on independent connections.
>>> >> >>
>>> >> >> I think it is something like this or a Load Balancer issue.
>>> >> > But as it is not easy to simulate using Browser, it is hard to
>>> prove.
>>> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > Note that tested application was not perfect as it had many
>>> Content
>>> >> >> > Encoding behaviour and other non fully standard issues.
>>> >> >> >
>>> >> >> > Thanks for you notes.
>>> >> >> > Regards
>>> >> >> > Philippe
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Cordialement.
>>> >> > Philippe Mouawad.
>>> >>
>>> >
>>> >
>>> > --
>>> > Cordialement.
>>> > Philippe Mouawad.
>>>
>>
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>
>>
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>
>>
>>
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
--
Cordialement.
Philippe Mouawad.
Re: Strange Behaviour with JavaImpl & Load Balancer
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello,
A little update about this topic, after further analysis it turned out
there was a bug in Load Balancer
which didn't handle correctly clients that use connection multiplexing.
Regarding this comment on Java Impl
"The API is best suited to single-threaded usage".
Don't you think we should change this ? it does not seem true and If it's
true, as JMeter is mainly used for Load Testing (ie multiple threads) it
seems to me weird, either this is wrong or the sentence is ambiguous, if
true we should remove it or limit it to functional testing.
As of my tests I think it works fine in multi threaded usage.
What's your opinion on this ?
Regards
Philippe
On Tue, Mar 26, 2013 at 7:42 AM, Philippe Mouawad <
philippe.mouawad@gmail.com> wrote:
> Hello sebb,
> Thanks for all your ideas.
> My answers below.
>
> Regards
> Philippe
>
> On Mon, Mar 25, 2013 at 9:38 PM, sebb <se...@gmail.com> wrote:
>
>> On 25 March 2013 19:38, Philippe Mouawad <ph...@gmail.com>
>> wrote:
>> > On Monday, March 25, 2013, sebb wrote:
>> >
>> >> On 25 March 2013 17:31, Philippe Mouawad <philippe.mouawad@gmail.com
>> <javascript:;>>
>> >> wrote:
>> >> > Hello,
>> >> > Thanks sebb, see my notes below.
>> >> >
>> >> > On Mon, Mar 25, 2013 at 6:21 PM, sebb <sebbaz@gmail.com<javascript:;>>
>> >> wrote:
>> >> >
>> >> >> On 25 March 2013 14:17, Philippe Mouawad <
>> philippe.mouawad@gmail.com<javascript:;>
>> >> >
>> >> >> wrote:
>> >> >> > Hello,
>> >> >> > I noticed a strange behaviour when Load Testing an application
>> >> recently.
>> >> >> >
>> >> >> > There was a Load Balancer in the middle:
>> >> >> >
>> >> >> > - Using Java Implementation I had around 8% of error
>> >> >>
>> >> >> What kinds of error?
>> >> >>
>> >> >> Session is lost, so it means load balancer redirects to other Apache
>> >> > ignoring Session Stickiness.
>> >>
>> >> What controls the session?
>> >
>> >
>> > A cookie one created by LB for session stickyness and one created by
>> Apache
>> > HttpServer
>> >
>> >>
>> >> Does it use cookies?
>> >
>> > Yes
>> >
>> >> Does it rely on the connection in any way?
>> >>
>> >> I don't think so. Hc do not have this issue.
>> >
>>
>> I don't think you can assume that, because JMeter's HC impl. uses
>> connections very differently.
>>
>> Maybe it assumes that requests with the same connection are related.
>> This cannot happen for HC (at least not in JMeter, because we don't
>> share connections).
>>
>> If you wanted to experiment further, you could use one of the HC
>> multi-threaded examples to test; but that might be a lot of work.
>>
>> >
>> >> >> > - Using HC3 or HC4 Implementations it was 0%
>> >> >> >
>> >> >> >
>> >> >> > I am sure it comes from Load Balancing as if it pointed on only
>> one
>> >> >> Apache
>> >> >> > then there was no issue.
>> >> >>
>> >> >> Note that the Java implementation may reuse connections between
>> threads.
>> >> >> That's one reason JMeter started using HC3 and HC4.
>> >> >>
>> >> >> > Now playing around with some Java System properties, I discovered
>> that
>> >> >> > setting *-Dhttp.keepAlive=false* fixed the error issue:
>> >> >>
>> >> >> That suggests it might be the connection reuse.
>> >> >>
>> >> > Agree
>> >> >
>> >> >>
>> >> >> > -
>> >> >> >
>> >> >>
>> >>
>> http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
>> >> >> >
>> >> >> > Another way to fix this was to uncheck KeepAlive checkbox in
>> >> samplers.
>> >> >>
>> >> >> Likewise, that should stop cross-thread connection sharing, as it
>> will
>> >> >> stop any sharing.
>> >> >>
>> >> >> >
>> >> >> > Now my question is:
>> >> >> >
>> >> >> > - Is this regular ? It seems strange to me that HC31 and HC4
>> impl
>> >> do
>> >> >> not
>> >> >> > have this behaviour
>> >> >> > - If yes, we should absolutely document this behaviour no ?
>> >> >>
>> >> >> I think it is; but it could perhaps be made more obvious.
>> >> >>
>> >> > Where is it ? Didn't see any mention of -Dhttp.keepAlive in docs
>> >>
>> >> The docs don't mention the property but they do mention why the Java
>> >> implementation is unsuitable; e.g.:
>> >>
>> >> >>
>> >> The Java HTTP implementation has some limitations:
>> >> * There is no control over how connections are re-used. When a
>> >> connection is released by JMeter, it may or may not be re-used by the
>> >> same thread.
>> >> <<
>> >>
>> >> Well reading this I wouldn't have thought we would face such failure.
>> >
>>
> Regarding this "The API is best suited to single-threaded usage". If it's
> true as JMeter is mainly used for Load Testing (ie multiple threads) it
> seems to me weird, either this is wrong or the sentence is ambiguous, if
> true we should remove it or limit it to functional testing.
>
>> >
>> >> By all means add details of the property if you think it would help.
>> >>
>> >> I will.
>> > I am not a big user of JAva Impl , had to as this same application
>> failed
>> > to record with HC some requests, ie when played by Http Server Proxy
>> they
>> > didn't return response returned by browser or Java Impl.
>>
>> That needs fixing.
>>
>> My issue is that it's a contract limited in time and I unfortunately
> don't have time at all to work on this diagnosis and fix, otherwise I would
> have :-)
> I think it comes from charset issues as application is not great at this
> but couldn't investigate further.
>
>
>> > But I must say it came to me as a bad surprise and would like if
>> keepalive
>> > is not handled correctly to kind of avoid these cases by adding it to
>> > startup script of not allowing keepAlive in this case.
>>
>> I don't think it's KeepAlive per se, otherwise HC would suffer.
>> AFAIK disabling KeepAlive forces the connection to be closed, so
>> obviously it cannot then be used by a different thread.
>> That should be possible to test by getting the threads to set a cookie
>> with the thread ID in it.
>> Or ideally a unique Cookie named after the thread, as that would
>> reveal any unintentional cookie sharing as there would be multiple
>> such cookies (I assume it cannot happen).
>>
>>
>
>> > But this may also only be related to this application and reveal an
>> > underlying issue in application.
>> > That's why I was asking about your experiences with Java Impl.
>> >
>> >
>> >> >>
>> >> >> > - Could it be a JDK 7 bug ?java version "1.7.0_13"
>> >> >>
>> >> >> No, it's by design.
>> >> >>
>> >> >> In theory the connnection sharing should be OK (otherwise I assume
>> >> >> Java would not do it), but maybe there is a subtle bug in the
>> >> >> application that relies on independent connections.
>> >> >>
>> >> >> I think it is something like this or a Load Balancer issue.
>> >> > But as it is not easy to simulate using Browser, it is hard to prove.
>> >> >
>> >> >> >
>> >> >> >
>> >> >> > Note that tested application was not perfect as it had many
>> Content
>> >> >> > Encoding behaviour and other non fully standard issues.
>> >> >> >
>> >> >> > Thanks for you notes.
>> >> >> > Regards
>> >> >> > Philippe
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Cordialement.
>> >> > Philippe Mouawad.
>> >>
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
>
--
Cordialement.
Philippe Mouawad.
Strange Behaviour with JavaImpl & Load Balancer
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello sebb,
Thanks for all your ideas.
My answers below.
Regards
Philippe
On Mon, Mar 25, 2013 at 9:38 PM, sebb <sebbaz@gmail.com <javascript:_e({},
'cvml', 'sebbaz@gmail.com');>> wrote:
> On 25 March 2013 19:38, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:_e({}, 'cvml', 'philippe.mouawad@gmail.com');>>
> wrote:
> > On Monday, March 25, 2013, sebb wrote:
> >
> >> On 25 March 2013 17:31, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:_e({}, 'cvml', 'philippe.mouawad@gmail.com');>
> <javascript:;>>
> >> wrote:
> >> > Hello,
> >> > Thanks sebb, see my notes below.
> >> >
> >> > On Mon, Mar 25, 2013 at 6:21 PM, sebb <sebbaz@gmail.com<javascript:_e({}, 'cvml', 'sebbaz@gmail.com');><javascript:;>>
> >> wrote:
> >> >
> >> >> On 25 March 2013 14:17, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:_e({}, 'cvml', 'philippe.mouawad@gmail.com');>
> <javascript:;>
> >> >
> >> >> wrote:
> >> >> > Hello,
> >> >> > I noticed a strange behaviour when Load Testing an application
> >> recently.
> >> >> >
> >> >> > There was a Load Balancer in the middle:
> >> >> >
> >> >> > - Using Java Implementation I had around 8% of error
> >> >>
> >> >> What kinds of error?
> >> >>
> >> >> Session is lost, so it means load balancer redirects to other Apache
> >> > ignoring Session Stickiness.
> >>
> >> What controls the session?
> >
> >
> > A cookie one created by LB for session stickyness and one created by
> Apache
> > HttpServer
> >
> >>
> >> Does it use cookies?
> >
> > Yes
> >
> >> Does it rely on the connection in any way?
> >>
> >> I don't think so. Hc do not have this issue.
> >
>
> I don't think you can assume that, because JMeter's HC impl. uses
> connections very differently.
>
> Maybe it assumes that requests with the same connection are related.
> This cannot happen for HC (at least not in JMeter, because we don't
> share connections).
>
> If you wanted to experiment further, you could use one of the HC
> multi-threaded examples to test; but that might be a lot of work.
>
> >
> >> >> > - Using HC3 or HC4 Implementations it was 0%
> >> >> >
> >> >> >
> >> >> > I am sure it comes from Load Balancing as if it pointed on only one
> >> >> Apache
> >> >> > then there was no issue.
> >> >>
> >> >> Note that the Java implementation may reuse connections between
> threads.
> >> >> That's one reason JMeter started using HC3 and HC4.
> >> >>
> >> >> > Now playing around with some Java System properties, I discovered
> that
> >> >> > setting *-Dhttp.keepAlive=false* fixed the error issue:
> >> >>
> >> >> That suggests it might be the connection reuse.
> >> >>
> >> > Agree
> >> >
> >> >>
> >> >> > -
> >> >> >
> >> >>
> >>
> http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
> >> >> >
> >> >> > Another way to fix this was to uncheck KeepAlive checkbox in
> >> samplers.
> >> >>
> >> >> Likewise, that should stop cross-thread connection sharing, as it
> will
> >> >> stop any sharing.
> >> >>
> >> >> >
> >> >> > Now my question is:
> >> >> >
> >> >> > - Is this regular ? It seems strange to me that HC31 and HC4
> impl
> >> do
> >> >> not
> >> >> > have this behaviour
> >> >> > - If yes, we should absolutely document this behaviour no ?
> >> >>
> >> >> I think it is; but it could perhaps be made more obvious.
> >> >>
> >> > Where is it ? Didn't see any mention of -Dhttp.keepAlive in docs
> >>
> >> The docs don't mention the property but they do mention why the Java
> >> implementation is unsuitable; e.g.:
> >>
> >> >>
> >> The Java HTTP implementation has some limitations:
> >> * There is no control over how connections are re-used. When a
> >> connection is released by JMeter, it may or may not be re-used by the
> >> same thread.
> >> <<
> >>
> >> Well reading this I wouldn't have thought we would face such failure.
> >
>
Regarding this "The API is best suited to single-threaded usage". If it's
true as JMeter is mainly used for Load Testing (ie multiple threads) it
seems to me weird, either this is wrong or the sentence is ambiguous, if
true we should remove it or limit it to functional testing.
> >
> >> By all means add details of the property if you think it would help.
> >>
> >> I will.
> > I am not a big user of JAva Impl , had to as this same application failed
> > to record with HC some requests, ie when played by Http Server Proxy they
> > didn't return response returned by browser or Java Impl.
>
> That needs fixing.
>
> My issue is that it's a contract limited in time and I unfortunately don't
have time at all to work on this diagnosis and fix, otherwise I would have
:-)
I think it comes from charset issues as application is not great at this
but couldn't investigate further.
> > But I must say it came to me as a bad surprise and would like if
> keepalive
> > is not handled correctly to kind of avoid these cases by adding it to
> > startup script of not allowing keepAlive in this case.
>
> I don't think it's KeepAlive per se, otherwise HC would suffer.
> AFAIK disabling KeepAlive forces the connection to be closed, so
> obviously it cannot then be used by a different thread.
> That should be possible to test by getting the threads to set a cookie
> with the thread ID in it.
> Or ideally a unique Cookie named after the thread, as that would
> reveal any unintentional cookie sharing as there would be multiple
> such cookies (I assume it cannot happen).
>
>
> > But this may also only be related to this application and reveal an
> > underlying issue in application.
> > That's why I was asking about your experiences with Java Impl.
> >
> >
> >> >>
> >> >> > - Could it be a JDK 7 bug ?java version "1.7.0_13"
> >> >>
> >> >> No, it's by design.
> >> >>
> >> >> In theory the connnection sharing should be OK (otherwise I assume
> >> >> Java would not do it), but maybe there is a subtle bug in the
> >> >> application that relies on independent connections.
> >> >>
> >> >> I think it is something like this or a Load Balancer issue.
> >> > But as it is not easy to simulate using Browser, it is hard to prove.
> >> >
> >> >> >
> >> >> >
> >> >> > Note that tested application was not perfect as it had many Content
> >> >> > Encoding behaviour and other non fully standard issues.
> >> >> >
> >> >> > Thanks for you notes.
> >> >> > Regards
> >> >> > Philippe
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Cordialement.
> >> > Philippe Mouawad.
> >>
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>
--
Cordialement.
Philippe Mouawad.
--
Cordialement.
Philippe Mouawad.
Re: Strange Behaviour with JavaImpl & Load Balancer
Posted by sebb <se...@gmail.com>.
On 25 March 2013 19:38, Philippe Mouawad <ph...@gmail.com> wrote:
> On Monday, March 25, 2013, sebb wrote:
>
>> On 25 March 2013 17:31, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>>
>> wrote:
>> > Hello,
>> > Thanks sebb, see my notes below.
>> >
>> > On Mon, Mar 25, 2013 at 6:21 PM, sebb <sebbaz@gmail.com <javascript:;>>
>> wrote:
>> >
>> >> On 25 March 2013 14:17, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>
>> >
>> >> wrote:
>> >> > Hello,
>> >> > I noticed a strange behaviour when Load Testing an application
>> recently.
>> >> >
>> >> > There was a Load Balancer in the middle:
>> >> >
>> >> > - Using Java Implementation I had around 8% of error
>> >>
>> >> What kinds of error?
>> >>
>> >> Session is lost, so it means load balancer redirects to other Apache
>> > ignoring Session Stickiness.
>>
>> What controls the session?
>
>
> A cookie one created by LB for session stickyness and one created by Apache
> HttpServer
>
>>
>> Does it use cookies?
>
> Yes
>
>> Does it rely on the connection in any way?
>>
>> I don't think so. Hc do not have this issue.
>
I don't think you can assume that, because JMeter's HC impl. uses
connections very differently.
Maybe it assumes that requests with the same connection are related.
This cannot happen for HC (at least not in JMeter, because we don't
share connections).
If you wanted to experiment further, you could use one of the HC
multi-threaded examples to test; but that might be a lot of work.
>
>> >> > - Using HC3 or HC4 Implementations it was 0%
>> >> >
>> >> >
>> >> > I am sure it comes from Load Balancing as if it pointed on only one
>> >> Apache
>> >> > then there was no issue.
>> >>
>> >> Note that the Java implementation may reuse connections between threads.
>> >> That's one reason JMeter started using HC3 and HC4.
>> >>
>> >> > Now playing around with some Java System properties, I discovered that
>> >> > setting *-Dhttp.keepAlive=false* fixed the error issue:
>> >>
>> >> That suggests it might be the connection reuse.
>> >>
>> > Agree
>> >
>> >>
>> >> > -
>> >> >
>> >>
>> http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
>> >> >
>> >> > Another way to fix this was to uncheck KeepAlive checkbox in
>> samplers.
>> >>
>> >> Likewise, that should stop cross-thread connection sharing, as it will
>> >> stop any sharing.
>> >>
>> >> >
>> >> > Now my question is:
>> >> >
>> >> > - Is this regular ? It seems strange to me that HC31 and HC4 impl
>> do
>> >> not
>> >> > have this behaviour
>> >> > - If yes, we should absolutely document this behaviour no ?
>> >>
>> >> I think it is; but it could perhaps be made more obvious.
>> >>
>> > Where is it ? Didn't see any mention of -Dhttp.keepAlive in docs
>>
>> The docs don't mention the property but they do mention why the Java
>> implementation is unsuitable; e.g.:
>>
>> >>
>> The Java HTTP implementation has some limitations:
>> * There is no control over how connections are re-used. When a
>> connection is released by JMeter, it may or may not be re-used by the
>> same thread.
>> <<
>>
>> Well reading this I wouldn't have thought we would face such failure.
>
>
>> By all means add details of the property if you think it would help.
>>
>> I will.
> I am not a big user of JAva Impl , had to as this same application failed
> to record with HC some requests, ie when played by Http Server Proxy they
> didn't return response returned by browser or Java Impl.
That needs fixing.
> But I must say it came to me as a bad surprise and would like if keepalive
> is not handled correctly to kind of avoid these cases by adding it to
> startup script of not allowing keepAlive in this case.
I don't think it's KeepAlive per se, otherwise HC would suffer.
AFAIK disabling KeepAlive forces the connection to be closed, so
obviously it cannot then be used by a different thread.
That should be possible to test by getting the threads to set a cookie
with the thread ID in it.
Or ideally a unique Cookie named after the thread, as that would
reveal any unintentional cookie sharing as there would be multiple
such cookies (I assume it cannot happen).
> But this may also only be related to this application and reveal an
> underlying issue in application.
> That's why I was asking about your experiences with Java Impl.
>
>
>> >>
>> >> > - Could it be a JDK 7 bug ?java version "1.7.0_13"
>> >>
>> >> No, it's by design.
>> >>
>> >> In theory the connnection sharing should be OK (otherwise I assume
>> >> Java would not do it), but maybe there is a subtle bug in the
>> >> application that relies on independent connections.
>> >>
>> >> I think it is something like this or a Load Balancer issue.
>> > But as it is not easy to simulate using Browser, it is hard to prove.
>> >
>> >> >
>> >> >
>> >> > Note that tested application was not perfect as it had many Content
>> >> > Encoding behaviour and other non fully standard issues.
>> >> >
>> >> > Thanks for you notes.
>> >> > Regards
>> >> > Philippe
>> >>
>> >
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.
Re: Strange Behaviour with JavaImpl & Load Balancer
Posted by Philippe Mouawad <ph...@gmail.com>.
On Monday, March 25, 2013, sebb wrote:
> On 25 March 2013 17:31, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>>
> wrote:
> > Hello,
> > Thanks sebb, see my notes below.
> >
> > On Mon, Mar 25, 2013 at 6:21 PM, sebb <sebbaz@gmail.com <javascript:;>>
> wrote:
> >
> >> On 25 March 2013 14:17, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>
> >
> >> wrote:
> >> > Hello,
> >> > I noticed a strange behaviour when Load Testing an application
> recently.
> >> >
> >> > There was a Load Balancer in the middle:
> >> >
> >> > - Using Java Implementation I had around 8% of error
> >>
> >> What kinds of error?
> >>
> >> Session is lost, so it means load balancer redirects to other Apache
> > ignoring Session Stickiness.
>
> What controls the session?
A cookie one created by LB for session stickyness and one created by Apache
HttpServer
>
> Does it use cookies?
Yes
> Does it rely on the connection in any way?
>
> I don't think so. Hc do not have this issue.
> >> > - Using HC3 or HC4 Implementations it was 0%
> >> >
> >> >
> >> > I am sure it comes from Load Balancing as if it pointed on only one
> >> Apache
> >> > then there was no issue.
> >>
> >> Note that the Java implementation may reuse connections between threads.
> >> That's one reason JMeter started using HC3 and HC4.
> >>
> >> > Now playing around with some Java System properties, I discovered that
> >> > setting *-Dhttp.keepAlive=false* fixed the error issue:
> >>
> >> That suggests it might be the connection reuse.
> >>
> > Agree
> >
> >>
> >> > -
> >> >
> >>
> http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
> >> >
> >> > Another way to fix this was to uncheck KeepAlive checkbox in
> samplers.
> >>
> >> Likewise, that should stop cross-thread connection sharing, as it will
> >> stop any sharing.
> >>
> >> >
> >> > Now my question is:
> >> >
> >> > - Is this regular ? It seems strange to me that HC31 and HC4 impl
> do
> >> not
> >> > have this behaviour
> >> > - If yes, we should absolutely document this behaviour no ?
> >>
> >> I think it is; but it could perhaps be made more obvious.
> >>
> > Where is it ? Didn't see any mention of -Dhttp.keepAlive in docs
>
> The docs don't mention the property but they do mention why the Java
> implementation is unsuitable; e.g.:
>
> >>
> The Java HTTP implementation has some limitations:
> * There is no control over how connections are re-used. When a
> connection is released by JMeter, it may or may not be re-used by the
> same thread.
> <<
>
> Well reading this I wouldn't have thought we would face such failure.
> By all means add details of the property if you think it would help.
>
> I will.
I am not a big user of JAva Impl , had to as this same application failed
to record with HC some requests, ie when played by Http Server Proxy they
didn't return response returned by browser or Java Impl.
But I must say it came to me as a bad surprise and would like if keepalive
is not handled correctly to kind of avoid these cases by adding it to
startup script of not allowing keepAlive in this case.
But this may also only be related to this application and reveal an
underlying issue in application.
That's why I was asking about your experiences with Java Impl.
> >>
> >> > - Could it be a JDK 7 bug ?java version "1.7.0_13"
> >>
> >> No, it's by design.
> >>
> >> In theory the connnection sharing should be OK (otherwise I assume
> >> Java would not do it), but maybe there is a subtle bug in the
> >> application that relies on independent connections.
> >>
> >> I think it is something like this or a Load Balancer issue.
> > But as it is not easy to simulate using Browser, it is hard to prove.
> >
> >> >
> >> >
> >> > Note that tested application was not perfect as it had many Content
> >> > Encoding behaviour and other non fully standard issues.
> >> >
> >> > Thanks for you notes.
> >> > Regards
> >> > Philippe
> >>
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>
--
Cordialement.
Philippe Mouawad.
Re: Strange Behaviour with JavaImpl & Load Balancer
Posted by sebb <se...@gmail.com>.
On 25 March 2013 17:31, Philippe Mouawad <ph...@gmail.com> wrote:
> Hello,
> Thanks sebb, see my notes below.
>
> On Mon, Mar 25, 2013 at 6:21 PM, sebb <se...@gmail.com> wrote:
>
>> On 25 March 2013 14:17, Philippe Mouawad <ph...@gmail.com>
>> wrote:
>> > Hello,
>> > I noticed a strange behaviour when Load Testing an application recently.
>> >
>> > There was a Load Balancer in the middle:
>> >
>> > - Using Java Implementation I had around 8% of error
>>
>> What kinds of error?
>>
>> Session is lost, so it means load balancer redirects to other Apache
> ignoring Session Stickiness.
What controls the session?
Does it use cookies?
Does it rely on the connection in any way?
>> > - Using HC3 or HC4 Implementations it was 0%
>> >
>> >
>> > I am sure it comes from Load Balancing as if it pointed on only one
>> Apache
>> > then there was no issue.
>>
>> Note that the Java implementation may reuse connections between threads.
>> That's one reason JMeter started using HC3 and HC4.
>>
>> > Now playing around with some Java System properties, I discovered that
>> > setting *-Dhttp.keepAlive=false* fixed the error issue:
>>
>> That suggests it might be the connection reuse.
>>
> Agree
>
>>
>> > -
>> >
>> http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
>> >
>> > Another way to fix this was to uncheck KeepAlive checkbox in samplers.
>>
>> Likewise, that should stop cross-thread connection sharing, as it will
>> stop any sharing.
>>
>> >
>> > Now my question is:
>> >
>> > - Is this regular ? It seems strange to me that HC31 and HC4 impl do
>> not
>> > have this behaviour
>> > - If yes, we should absolutely document this behaviour no ?
>>
>> I think it is; but it could perhaps be made more obvious.
>>
> Where is it ? Didn't see any mention of -Dhttp.keepAlive in docs
The docs don't mention the property but they do mention why the Java
implementation is unsuitable; e.g.:
>>
The Java HTTP implementation has some limitations:
* There is no control over how connections are re-used. When a
connection is released by JMeter, it may or may not be re-used by the
same thread.
<<
By all means add details of the property if you think it would help.
>>
>> > - Could it be a JDK 7 bug ?java version "1.7.0_13"
>>
>> No, it's by design.
>>
>> In theory the connnection sharing should be OK (otherwise I assume
>> Java would not do it), but maybe there is a subtle bug in the
>> application that relies on independent connections.
>>
>> I think it is something like this or a Load Balancer issue.
> But as it is not easy to simulate using Browser, it is hard to prove.
>
>> >
>> >
>> > Note that tested application was not perfect as it had many Content
>> > Encoding behaviour and other non fully standard issues.
>> >
>> > Thanks for you notes.
>> > Regards
>> > Philippe
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
Re: Strange Behaviour with JavaImpl & Load Balancer
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello,
Thanks sebb, see my notes below.
On Mon, Mar 25, 2013 at 6:21 PM, sebb <se...@gmail.com> wrote:
> On 25 March 2013 14:17, Philippe Mouawad <ph...@gmail.com>
> wrote:
> > Hello,
> > I noticed a strange behaviour when Load Testing an application recently.
> >
> > There was a Load Balancer in the middle:
> >
> > - Using Java Implementation I had around 8% of error
>
> What kinds of error?
>
> Session is lost, so it means load balancer redirects to other Apache
ignoring Session Stickiness.
> > - Using HC3 or HC4 Implementations it was 0%
> >
> >
> > I am sure it comes from Load Balancing as if it pointed on only one
> Apache
> > then there was no issue.
>
> Note that the Java implementation may reuse connections between threads.
> That's one reason JMeter started using HC3 and HC4.
>
> > Now playing around with some Java System properties, I discovered that
> > setting *-Dhttp.keepAlive=false* fixed the error issue:
>
> That suggests it might be the connection reuse.
>
Agree
>
> > -
> >
> http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
> >
> > Another way to fix this was to uncheck KeepAlive checkbox in samplers.
>
> Likewise, that should stop cross-thread connection sharing, as it will
> stop any sharing.
>
> >
> > Now my question is:
> >
> > - Is this regular ? It seems strange to me that HC31 and HC4 impl do
> not
> > have this behaviour
> > - If yes, we should absolutely document this behaviour no ?
>
> I think it is; but it could perhaps be made more obvious.
>
Where is it ? Didn't see any mention of -Dhttp.keepAlive in docs
>
> > - Could it be a JDK 7 bug ?java version "1.7.0_13"
>
> No, it's by design.
>
> In theory the connnection sharing should be OK (otherwise I assume
> Java would not do it), but maybe there is a subtle bug in the
> application that relies on independent connections.
>
> I think it is something like this or a Load Balancer issue.
But as it is not easy to simulate using Browser, it is hard to prove.
> >
> >
> > Note that tested application was not perfect as it had many Content
> > Encoding behaviour and other non fully standard issues.
> >
> > Thanks for you notes.
> > Regards
> > Philippe
>
--
Cordialement.
Philippe Mouawad.
Re: Strange Behaviour with JavaImpl & Load Balancer
Posted by sebb <se...@gmail.com>.
On 25 March 2013 14:17, Philippe Mouawad <ph...@gmail.com> wrote:
> Hello,
> I noticed a strange behaviour when Load Testing an application recently.
>
> There was a Load Balancer in the middle:
>
> - Using Java Implementation I had around 8% of error
What kinds of error?
> - Using HC3 or HC4 Implementations it was 0%
>
>
> I am sure it comes from Load Balancing as if it pointed on only one Apache
> then there was no issue.
Note that the Java implementation may reuse connections between threads.
That's one reason JMeter started using HC3 and HC4.
> Now playing around with some Java System properties, I discovered that
> setting *-Dhttp.keepAlive=false* fixed the error issue:
That suggests it might be the connection reuse.
> -
> http://docs.oracle.com/javase/6/docs/technotes/guides/net/http-keepalive.html
>
> Another way to fix this was to uncheck KeepAlive checkbox in samplers.
Likewise, that should stop cross-thread connection sharing, as it will
stop any sharing.
>
> Now my question is:
>
> - Is this regular ? It seems strange to me that HC31 and HC4 impl do not
> have this behaviour
> - If yes, we should absolutely document this behaviour no ?
I think it is; but it could perhaps be made more obvious.
> - Could it be a JDK 7 bug ?java version "1.7.0_13"
No, it's by design.
In theory the connnection sharing should be OK (otherwise I assume
Java would not do it), but maybe there is a subtle bug in the
application that relies on independent connections.
>
>
> Note that tested application was not perfect as it had many Content
> Encoding behaviour and other non fully standard issues.
>
> Thanks for you notes.
> Regards
> Philippe