You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by "Warren, Jared S" <js...@jcp.com> on 2014/01/16 17:28:32 UTC
CXF Client performance troubleshooting
Hey, all -
I'm facing an interesting challenge.
I'm using CXF as a web services client, calling a server hosted on a different platform.
I've instrumented my code around the call to the CXF port and observe that 10 - 20% of the requests perform very poorly (a "good" response time is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few requests that fall between "good" and "bad").
The server-side log web access log shows that all requests response times were "good" (100 - 200ms). For those requests that the server and client both think were "good", there is very little latency between the client measurements and the server measurements (only a few ms additional time in the CXF client than the server thinks the request took). Bandwidth on the network segments between the two is nowhere close to saturated. Besides that, I don't think network latency would inject SECONDS (perhaps milliseconds).
This only occurs under very heavy load, so packet sniffing to figure out which side of the wire the problem is on would be extremely arduous (plus, I don't have easy access to sniff the relevant network segments).
So...here's my question....
Is there any performance logging I can enable in CXF so that I can see "inside" CXF and try to push my visibility "closer to the wire"?
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
RE: CXF Client performance troubleshooting
Posted by "Warren, Jared S" <js...@jcp.com>.
We've confirmed that the poor performance doesn't line up with GC - that was a good theory, though.
I've been reviewing the CXF source, though, and noticed that HTTPConduit seems to support "retransmits". The behavior we see could be explained by that: somehow, the original request could be failing (for some reason) before it hits the server, so the initial request never shows up in the server logs. If we sit in a retransmit loop for some time (inside CXF) before finally reaching the server, that would explain long response times recorded in the client code, and short response times logged in the server (and, the number of requests would match).
Does anyone know enough about the retransmit logic in org.apache.cxf.transport.http.HttpConduit.WrappedOutputStream enough to briefly explain it?
-----Original Message-----
From: Andrei Shakirin [mailto:ashakirin@talend.com]
Sent: Thursday, January 16, 2014 10:55 AM
To: users@cxf.apache.org
Subject: RE: CXF Client performance troubleshooting
Hi,
Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests.
You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain.
Curious how your performance results correlates with these ones: http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-platform.html
Regards,
Andrei.
> -----Original Message-----
> From: Warren, Jared S [mailto:jswarren@jcp.com]
> Sent: Donnerstag, 16. Januar 2014 17:29
> To: users@cxf.apache.org
> Subject: CXF Client performance troubleshooting
>
> Hey, all -
>
> I'm facing an interesting challenge.
>
> I'm using CXF as a web services client, calling a server hosted on a
> different platform.
>
> I've instrumented my code around the call to the CXF port and observe
> that
> 10 - 20% of the requests perform very poorly (a "good" response time
> is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few
> requests that fall between "good" and "bad").
>
> The server-side log web access log shows that all requests response
> times were "good" (100 - 200ms). For those requests that the server
> and client both think were "good", there is very little latency
> between the client measurements and the server measurements (only a
> few ms additional time in the CXF client than the server thinks the
> request took). Bandwidth on the network segments between the two is
> nowhere close to saturated. Besides that, I don't think network
> latency would inject SECONDS (perhaps milliseconds).
>
> This only occurs under very heavy load, so packet sniffing to figure
> out which side of the wire the problem is on would be extremely
> arduous (plus, I don't have easy access to sniff the relevant network segments).
>
> So...here's my question....
>
> Is there any performance logging I can enable in CXF so that I can see "inside"
> CXF and try to push my visibility "closer to the wire"?
>
>
>
>
>
>
>
> The information transmitted is intended only for the person or entity
> to which it is addressed and may contain confidential and/or privileged material.
> If the reader of this message is not the intended recipient, you are
> hereby notified that your access is unauthorized, and any review,
> dissemination, distribution or copying of this message including any
> attachments is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete the material from any computer.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
Re: CXF Client performance troubleshooting
Posted by Dennis Sosnoski <dm...@sosnoski.com>.
The IBM JSSE supports these, too:
http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/index.jsp?topic=%2Fcom.ibm.java.security.component.60.doc%2Fsecurity-component%2Fjsse2Docs%2Fdebug.html
- Dennis
On 01/21/2014 06:43 AM, Warren, Jared S wrote:
> Unfortunately, we're on WebSphere with the IBM JDK....I'm assuming these probably won't work there... ungh...
>
> -----Original Message-----
> From: Daniel Kulp [mailto:dkulp@apache.org]
> Sent: Monday, January 20, 2014 11:42 AM
> To: users@cxf.apache.org; Warren, Jared S
> Subject: Re: CXF Client performance troubleshooting
>
>
> Only thing I can think of is the -Djavax.net.debug=all or -Djavax.net.debug=ssl system properties. Debugging SSL stuff sucks. Good luck. :-(
>
> Dan
>
>
> On Jan 20, 2014, at 12:31 PM, Warren, Jared S <js...@jcp.com> wrote:
>
>> A further update:
>>
>> We had been running the performance update via HTTPS. We got some tcpdumps taken during the test to attempt to isolate the issue to one side of the wire (or the other). In the process, we demonstrated that our connection pooling/keepalive was working, which made it difficult/impossible to use the encrypted tcpdump, because we couldn't tell whether "idle" periods on a thread were due to keepalive reuse, or were actual performance issues.
>>
>> So, we switched to using unencrypted HTTP. Viola - our performance gremlin disappeared (we've now executed the load test 3 times to verify that we didn't just get lucky somehow).
>>
>> So...now we're off to troubleshoot the HTTPS/SSL configuration.
>>
>> Any tips here before I dive in myself?
>>
>>
>>
>> -----Original Message-----
>> From: Warren, Jared S [mailto:jswarren@jcp.com]
>> Sent: Thursday, January 16, 2014 12:22 PM
>> To: users@cxf.apache.org
>> Subject: RE: CXF Client performance troubleshooting
>>
>> FWIW - we're on CXF 2.7.8.
>>
>> After cranking up some logging, we can see that the delays come between these two log messages (note the 9 second delay between the timestamps):
>>
>> 11:27:29.128 AM 2014-01-16 11:27:29,128 DEBUG
>> [org.apache.cxf.transport.http.HTTPConduit] -
>> [P14263-4866400664992057849] - Sending POST Message with Headers to
>> https://xxxxxxx.com/reward.asmx?wsdl Conduit
>> :{http://xxxxxxxxx.com/webservices/}RewardSoap.http-conduit
>>
>> 11:27:38.313 AM 2014-01-16 11:27:38,313 DEBUG
>> [org.apache.cxf.endpoint.ClientImpl] - [P14263-4866400664992057849] -
>> Interceptors contributed by bus:
>> [com.xxx.cdi.integration.interceptor.MDCEnablingInterceptor@712d08fa,
>> com.xxx.cdi.integration.interceptor.ResponseTimeInInterceptor@c30e41f7
>> ,
>> com.xxx.cdi.integration.interceptor.SniffingInputStreamInInterceptor@a
>> 7cb0822,
>> com.xxx.cdi.integration.interceptor.DOMGeneratorInInterceptor@352167e8
>> , org.apache.cxf.ws.policy.PolicyInInterceptor@3166354]
>>
>> I don't know the CXF codebase well enough to interpret this (I'm downloading source and setting up a project shortly). Can anyone help out with an interpretation? The "com.xxx.cdi.integration.interceptor.*" are custom interceptors written by my team. Is this an indicator they're performing poorly?
>>
>> (we thought we had previously proved that they were performing great...but perhaps that was incorrect).
>>
>> Thanks!
>> jared
>>
>> -----Original Message-----
>> From: Andrei Shakirin [mailto:ashakirin@talend.com]
>> Sent: Thursday, January 16, 2014 10:55 AM
>> To: users@cxf.apache.org
>> Subject: RE: CXF Client performance troubleshooting
>>
>> Hi,
>>
>> Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests.
>> You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain.
>>
>> Curious how your performance results correlates with these ones:
>> http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-pl
>> atform.html
>>
>> Regards,
>> Andrei.
>>
>>> -----Original Message-----
>>> From: Warren, Jared S [mailto:jswarren@jcp.com]
>>> Sent: Donnerstag, 16. Januar 2014 17:29
>>> To: users@cxf.apache.org
>>> Subject: CXF Client performance troubleshooting
>>>
>>> Hey, all -
>>>
>>> I'm facing an interesting challenge.
>>>
>>> I'm using CXF as a web services client, calling a server hosted on a
>>> different platform.
>>>
>>> I've instrumented my code around the call to the CXF port and observe
>>> that
>>> 10 - 20% of the requests perform very poorly (a "good" response time
>>> is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few
>>> requests that fall between "good" and "bad").
>>>
>>> The server-side log web access log shows that all requests response
>>> times were "good" (100 - 200ms). For those requests that the server
>>> and client both think were "good", there is very little latency
>>> between the client measurements and the server measurements (only a
>>> few ms additional time in the CXF client than the server thinks the
>>> request took). Bandwidth on the network segments between the two is
>>> nowhere close to saturated. Besides that, I don't think network
>>> latency would inject SECONDS (perhaps milliseconds).
>>>
>>> This only occurs under very heavy load, so packet sniffing to figure
>>> out which side of the wire the problem is on would be extremely
>>> arduous (plus, I don't have easy access to sniff the relevant network segments).
>>>
>>> So...here's my question....
>>>
>>> Is there any performance logging I can enable in CXF so that I can see "inside"
>>> CXF and try to push my visibility "closer to the wire"?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The information transmitted is intended only for the person or entity
>>> to which it is addressed and may contain confidential and/or privileged material.
>>> If the reader of this message is not the intended recipient, you are
>>> hereby notified that your access is unauthorized, and any review,
>>> dissemination, distribution or copying of this message including any
>>> attachments is strictly prohibited. If you are not the intended
>>> recipient, please contact the sender and delete the material from any computer.
>> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
>> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
> --
> Daniel Kulp
> dkulp@apache.org - http://dankulp.com/blog Talend Community Coder - http://coders.talend.com
>
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
>
RE: CXF Client performance troubleshooting
Posted by "Warren, Jared S" <js...@jcp.com>.
Unfortunately, we're on WebSphere with the IBM JDK....I'm assuming these probably won't work there... ungh...
-----Original Message-----
From: Daniel Kulp [mailto:dkulp@apache.org]
Sent: Monday, January 20, 2014 11:42 AM
To: users@cxf.apache.org; Warren, Jared S
Subject: Re: CXF Client performance troubleshooting
Only thing I can think of is the -Djavax.net.debug=all or -Djavax.net.debug=ssl system properties. Debugging SSL stuff sucks. Good luck. :-(
Dan
On Jan 20, 2014, at 12:31 PM, Warren, Jared S <js...@jcp.com> wrote:
> A further update:
>
> We had been running the performance update via HTTPS. We got some tcpdumps taken during the test to attempt to isolate the issue to one side of the wire (or the other). In the process, we demonstrated that our connection pooling/keepalive was working, which made it difficult/impossible to use the encrypted tcpdump, because we couldn't tell whether "idle" periods on a thread were due to keepalive reuse, or were actual performance issues.
>
> So, we switched to using unencrypted HTTP. Viola - our performance gremlin disappeared (we've now executed the load test 3 times to verify that we didn't just get lucky somehow).
>
> So...now we're off to troubleshoot the HTTPS/SSL configuration.
>
> Any tips here before I dive in myself?
>
>
>
> -----Original Message-----
> From: Warren, Jared S [mailto:jswarren@jcp.com]
> Sent: Thursday, January 16, 2014 12:22 PM
> To: users@cxf.apache.org
> Subject: RE: CXF Client performance troubleshooting
>
> FWIW - we're on CXF 2.7.8.
>
> After cranking up some logging, we can see that the delays come between these two log messages (note the 9 second delay between the timestamps):
>
> 11:27:29.128 AM 2014-01-16 11:27:29,128 DEBUG
> [org.apache.cxf.transport.http.HTTPConduit] -
> [P14263-4866400664992057849] - Sending POST Message with Headers to
> https://xxxxxxx.com/reward.asmx?wsdl Conduit
> :{http://xxxxxxxxx.com/webservices/}RewardSoap.http-conduit
>
> 11:27:38.313 AM 2014-01-16 11:27:38,313 DEBUG
> [org.apache.cxf.endpoint.ClientImpl] - [P14263-4866400664992057849] -
> Interceptors contributed by bus:
> [com.xxx.cdi.integration.interceptor.MDCEnablingInterceptor@712d08fa,
> com.xxx.cdi.integration.interceptor.ResponseTimeInInterceptor@c30e41f7
> ,
> com.xxx.cdi.integration.interceptor.SniffingInputStreamInInterceptor@a
> 7cb0822,
> com.xxx.cdi.integration.interceptor.DOMGeneratorInInterceptor@352167e8
> , org.apache.cxf.ws.policy.PolicyInInterceptor@3166354]
>
> I don't know the CXF codebase well enough to interpret this (I'm downloading source and setting up a project shortly). Can anyone help out with an interpretation? The "com.xxx.cdi.integration.interceptor.*" are custom interceptors written by my team. Is this an indicator they're performing poorly?
>
> (we thought we had previously proved that they were performing great...but perhaps that was incorrect).
>
> Thanks!
> jared
>
> -----Original Message-----
> From: Andrei Shakirin [mailto:ashakirin@talend.com]
> Sent: Thursday, January 16, 2014 10:55 AM
> To: users@cxf.apache.org
> Subject: RE: CXF Client performance troubleshooting
>
> Hi,
>
> Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests.
> You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain.
>
> Curious how your performance results correlates with these ones:
> http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-pl
> atform.html
>
> Regards,
> Andrei.
>
>> -----Original Message-----
>> From: Warren, Jared S [mailto:jswarren@jcp.com]
>> Sent: Donnerstag, 16. Januar 2014 17:29
>> To: users@cxf.apache.org
>> Subject: CXF Client performance troubleshooting
>>
>> Hey, all -
>>
>> I'm facing an interesting challenge.
>>
>> I'm using CXF as a web services client, calling a server hosted on a
>> different platform.
>>
>> I've instrumented my code around the call to the CXF port and observe
>> that
>> 10 - 20% of the requests perform very poorly (a "good" response time
>> is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few
>> requests that fall between "good" and "bad").
>>
>> The server-side log web access log shows that all requests response
>> times were "good" (100 - 200ms). For those requests that the server
>> and client both think were "good", there is very little latency
>> between the client measurements and the server measurements (only a
>> few ms additional time in the CXF client than the server thinks the
>> request took). Bandwidth on the network segments between the two is
>> nowhere close to saturated. Besides that, I don't think network
>> latency would inject SECONDS (perhaps milliseconds).
>>
>> This only occurs under very heavy load, so packet sniffing to figure
>> out which side of the wire the problem is on would be extremely
>> arduous (plus, I don't have easy access to sniff the relevant network segments).
>>
>> So...here's my question....
>>
>> Is there any performance logging I can enable in CXF so that I can see "inside"
>> CXF and try to push my visibility "closer to the wire"?
>>
>>
>>
>>
>>
>>
>>
>> The information transmitted is intended only for the person or entity
>> to which it is addressed and may contain confidential and/or privileged material.
>> If the reader of this message is not the intended recipient, you are
>> hereby notified that your access is unauthorized, and any review,
>> dissemination, distribution or copying of this message including any
>> attachments is strictly prohibited. If you are not the intended
>> recipient, please contact the sender and delete the material from any computer.
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
--
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog Talend Community Coder - http://coders.talend.com
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
Re: CXF Client performance troubleshooting
Posted by Daniel Kulp <dk...@apache.org>.
Only thing I can think of is the -Djavax.net.debug=all or -Djavax.net.debug=ssl system properties. Debugging SSL stuff sucks. Good luck. :-(
Dan
On Jan 20, 2014, at 12:31 PM, Warren, Jared S <js...@jcp.com> wrote:
> A further update:
>
> We had been running the performance update via HTTPS. We got some tcpdumps taken during the test to attempt to isolate the issue to one side of the wire (or the other). In the process, we demonstrated that our connection pooling/keepalive was working, which made it difficult/impossible to use the encrypted tcpdump, because we couldn't tell whether "idle" periods on a thread were due to keepalive reuse, or were actual performance issues.
>
> So, we switched to using unencrypted HTTP. Viola - our performance gremlin disappeared (we've now executed the load test 3 times to verify that we didn't just get lucky somehow).
>
> So...now we're off to troubleshoot the HTTPS/SSL configuration.
>
> Any tips here before I dive in myself?
>
>
>
> -----Original Message-----
> From: Warren, Jared S [mailto:jswarren@jcp.com]
> Sent: Thursday, January 16, 2014 12:22 PM
> To: users@cxf.apache.org
> Subject: RE: CXF Client performance troubleshooting
>
> FWIW - we're on CXF 2.7.8.
>
> After cranking up some logging, we can see that the delays come between these two log messages (note the 9 second delay between the timestamps):
>
> 11:27:29.128 AM 2014-01-16 11:27:29,128 DEBUG [org.apache.cxf.transport.http.HTTPConduit] - [P14263-4866400664992057849] - Sending POST Message with Headers to https://xxxxxxx.com/reward.asmx?wsdl Conduit :{http://xxxxxxxxx.com/webservices/}RewardSoap.http-conduit
>
> 11:27:38.313 AM 2014-01-16 11:27:38,313 DEBUG [org.apache.cxf.endpoint.ClientImpl] - [P14263-4866400664992057849] - Interceptors contributed by bus: [com.xxx.cdi.integration.interceptor.MDCEnablingInterceptor@712d08fa, com.xxx.cdi.integration.interceptor.ResponseTimeInInterceptor@c30e41f7, com.xxx.cdi.integration.interceptor.SniffingInputStreamInInterceptor@a7cb0822, com.xxx.cdi.integration.interceptor.DOMGeneratorInInterceptor@352167e8, org.apache.cxf.ws.policy.PolicyInInterceptor@3166354]
>
> I don't know the CXF codebase well enough to interpret this (I'm downloading source and setting up a project shortly). Can anyone help out with an interpretation? The "com.xxx.cdi.integration.interceptor.*" are custom interceptors written by my team. Is this an indicator they're performing poorly?
>
> (we thought we had previously proved that they were performing great...but perhaps that was incorrect).
>
> Thanks!
> jared
>
> -----Original Message-----
> From: Andrei Shakirin [mailto:ashakirin@talend.com]
> Sent: Thursday, January 16, 2014 10:55 AM
> To: users@cxf.apache.org
> Subject: RE: CXF Client performance troubleshooting
>
> Hi,
>
> Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests.
> You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain.
>
> Curious how your performance results correlates with these ones: http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-platform.html
>
> Regards,
> Andrei.
>
>> -----Original Message-----
>> From: Warren, Jared S [mailto:jswarren@jcp.com]
>> Sent: Donnerstag, 16. Januar 2014 17:29
>> To: users@cxf.apache.org
>> Subject: CXF Client performance troubleshooting
>>
>> Hey, all -
>>
>> I'm facing an interesting challenge.
>>
>> I'm using CXF as a web services client, calling a server hosted on a
>> different platform.
>>
>> I've instrumented my code around the call to the CXF port and observe
>> that
>> 10 - 20% of the requests perform very poorly (a "good" response time
>> is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few
>> requests that fall between "good" and "bad").
>>
>> The server-side log web access log shows that all requests response
>> times were "good" (100 - 200ms). For those requests that the server
>> and client both think were "good", there is very little latency
>> between the client measurements and the server measurements (only a
>> few ms additional time in the CXF client than the server thinks the
>> request took). Bandwidth on the network segments between the two is
>> nowhere close to saturated. Besides that, I don't think network
>> latency would inject SECONDS (perhaps milliseconds).
>>
>> This only occurs under very heavy load, so packet sniffing to figure
>> out which side of the wire the problem is on would be extremely
>> arduous (plus, I don't have easy access to sniff the relevant network segments).
>>
>> So...here's my question....
>>
>> Is there any performance logging I can enable in CXF so that I can see "inside"
>> CXF and try to push my visibility "closer to the wire"?
>>
>>
>>
>>
>>
>>
>>
>> The information transmitted is intended only for the person or entity
>> to which it is addressed and may contain confidential and/or privileged material.
>> If the reader of this message is not the intended recipient, you are
>> hereby notified that your access is unauthorized, and any review,
>> dissemination, distribution or copying of this message including any
>> attachments is strictly prohibited. If you are not the intended
>> recipient, please contact the sender and delete the material from any computer.
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
--
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com
RE: CXF Client performance troubleshooting
Posted by "Warren, Jared S" <js...@jcp.com>.
A further update:
We had been running the performance update via HTTPS. We got some tcpdumps taken during the test to attempt to isolate the issue to one side of the wire (or the other). In the process, we demonstrated that our connection pooling/keepalive was working, which made it difficult/impossible to use the encrypted tcpdump, because we couldn't tell whether "idle" periods on a thread were due to keepalive reuse, or were actual performance issues.
So, we switched to using unencrypted HTTP. Viola - our performance gremlin disappeared (we've now executed the load test 3 times to verify that we didn't just get lucky somehow).
So...now we're off to troubleshoot the HTTPS/SSL configuration.
Any tips here before I dive in myself?
-----Original Message-----
From: Warren, Jared S [mailto:jswarren@jcp.com]
Sent: Thursday, January 16, 2014 12:22 PM
To: users@cxf.apache.org
Subject: RE: CXF Client performance troubleshooting
FWIW - we're on CXF 2.7.8.
After cranking up some logging, we can see that the delays come between these two log messages (note the 9 second delay between the timestamps):
11:27:29.128 AM 2014-01-16 11:27:29,128 DEBUG [org.apache.cxf.transport.http.HTTPConduit] - [P14263-4866400664992057849] - Sending POST Message with Headers to https://xxxxxxx.com/reward.asmx?wsdl Conduit :{http://xxxxxxxxx.com/webservices/}RewardSoap.http-conduit
11:27:38.313 AM 2014-01-16 11:27:38,313 DEBUG [org.apache.cxf.endpoint.ClientImpl] - [P14263-4866400664992057849] - Interceptors contributed by bus: [com.xxx.cdi.integration.interceptor.MDCEnablingInterceptor@712d08fa, com.xxx.cdi.integration.interceptor.ResponseTimeInInterceptor@c30e41f7, com.xxx.cdi.integration.interceptor.SniffingInputStreamInInterceptor@a7cb0822, com.xxx.cdi.integration.interceptor.DOMGeneratorInInterceptor@352167e8, org.apache.cxf.ws.policy.PolicyInInterceptor@3166354]
I don't know the CXF codebase well enough to interpret this (I'm downloading source and setting up a project shortly). Can anyone help out with an interpretation? The "com.xxx.cdi.integration.interceptor.*" are custom interceptors written by my team. Is this an indicator they're performing poorly?
(we thought we had previously proved that they were performing great...but perhaps that was incorrect).
Thanks!
jared
-----Original Message-----
From: Andrei Shakirin [mailto:ashakirin@talend.com]
Sent: Thursday, January 16, 2014 10:55 AM
To: users@cxf.apache.org
Subject: RE: CXF Client performance troubleshooting
Hi,
Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests.
You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain.
Curious how your performance results correlates with these ones: http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-platform.html
Regards,
Andrei.
> -----Original Message-----
> From: Warren, Jared S [mailto:jswarren@jcp.com]
> Sent: Donnerstag, 16. Januar 2014 17:29
> To: users@cxf.apache.org
> Subject: CXF Client performance troubleshooting
>
> Hey, all -
>
> I'm facing an interesting challenge.
>
> I'm using CXF as a web services client, calling a server hosted on a
> different platform.
>
> I've instrumented my code around the call to the CXF port and observe
> that
> 10 - 20% of the requests perform very poorly (a "good" response time
> is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few
> requests that fall between "good" and "bad").
>
> The server-side log web access log shows that all requests response
> times were "good" (100 - 200ms). For those requests that the server
> and client both think were "good", there is very little latency
> between the client measurements and the server measurements (only a
> few ms additional time in the CXF client than the server thinks the
> request took). Bandwidth on the network segments between the two is
> nowhere close to saturated. Besides that, I don't think network
> latency would inject SECONDS (perhaps milliseconds).
>
> This only occurs under very heavy load, so packet sniffing to figure
> out which side of the wire the problem is on would be extremely
> arduous (plus, I don't have easy access to sniff the relevant network segments).
>
> So...here's my question....
>
> Is there any performance logging I can enable in CXF so that I can see "inside"
> CXF and try to push my visibility "closer to the wire"?
>
>
>
>
>
>
>
> The information transmitted is intended only for the person or entity
> to which it is addressed and may contain confidential and/or privileged material.
> If the reader of this message is not the intended recipient, you are
> hereby notified that your access is unauthorized, and any review,
> dissemination, distribution or copying of this message including any
> attachments is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete the material from any computer.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
RE: CXF Client performance troubleshooting
Posted by "Warren, Jared S" <js...@jcp.com>.
FWIW - we're on CXF 2.7.8.
After cranking up some logging, we can see that the delays come between these two log messages (note the 9 second delay between the timestamps):
11:27:29.128 AM 2014-01-16 11:27:29,128 DEBUG [org.apache.cxf.transport.http.HTTPConduit] - [P14263-4866400664992057849] - Sending POST Message with Headers to https://xxxxxxx.com/reward.asmx?wsdl Conduit :{http://xxxxxxxxx.com/webservices/}RewardSoap.http-conduit
11:27:38.313 AM 2014-01-16 11:27:38,313 DEBUG [org.apache.cxf.endpoint.ClientImpl] - [P14263-4866400664992057849] - Interceptors contributed by bus: [com.xxx.cdi.integration.interceptor.MDCEnablingInterceptor@712d08fa, com.xxx.cdi.integration.interceptor.ResponseTimeInInterceptor@c30e41f7, com.xxx.cdi.integration.interceptor.SniffingInputStreamInInterceptor@a7cb0822, com.xxx.cdi.integration.interceptor.DOMGeneratorInInterceptor@352167e8, org.apache.cxf.ws.policy.PolicyInInterceptor@3166354]
I don't know the CXF codebase well enough to interpret this (I'm downloading source and setting up a project shortly). Can anyone help out with an interpretation? The "com.xxx.cdi.integration.interceptor.*" are custom interceptors written by my team. Is this an indicator they're performing poorly?
(we thought we had previously proved that they were performing great...but perhaps that was incorrect).
Thanks!
jared
-----Original Message-----
From: Andrei Shakirin [mailto:ashakirin@talend.com]
Sent: Thursday, January 16, 2014 10:55 AM
To: users@cxf.apache.org
Subject: RE: CXF Client performance troubleshooting
Hi,
Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests.
You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain.
Curious how your performance results correlates with these ones: http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-platform.html
Regards,
Andrei.
> -----Original Message-----
> From: Warren, Jared S [mailto:jswarren@jcp.com]
> Sent: Donnerstag, 16. Januar 2014 17:29
> To: users@cxf.apache.org
> Subject: CXF Client performance troubleshooting
>
> Hey, all -
>
> I'm facing an interesting challenge.
>
> I'm using CXF as a web services client, calling a server hosted on a
> different platform.
>
> I've instrumented my code around the call to the CXF port and observe
> that
> 10 - 20% of the requests perform very poorly (a "good" response time
> is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few
> requests that fall between "good" and "bad").
>
> The server-side log web access log shows that all requests response
> times were "good" (100 - 200ms). For those requests that the server
> and client both think were "good", there is very little latency
> between the client measurements and the server measurements (only a
> few ms additional time in the CXF client than the server thinks the
> request took). Bandwidth on the network segments between the two is
> nowhere close to saturated. Besides that, I don't think network
> latency would inject SECONDS (perhaps milliseconds).
>
> This only occurs under very heavy load, so packet sniffing to figure
> out which side of the wire the problem is on would be extremely
> arduous (plus, I don't have easy access to sniff the relevant network segments).
>
> So...here's my question....
>
> Is there any performance logging I can enable in CXF so that I can see "inside"
> CXF and try to push my visibility "closer to the wire"?
>
>
>
>
>
>
>
> The information transmitted is intended only for the person or entity
> to which it is addressed and may contain confidential and/or privileged material.
> If the reader of this message is not the intended recipient, you are
> hereby notified that your access is unauthorized, and any review,
> dissemination, distribution or copying of this message including any
> attachments is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete the material from any computer.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.
RE: CXF Client performance troubleshooting
Posted by Andrei Shakirin <as...@talend.com>.
Hi,
Interesting, as a guess the performance degradation in some moments can be caused by JVM GC. I would suggest to run JVM monitor in parallel and look is heap size will be decreased in the moments of "bad" requests.
You can try to activate CXF Logging feature (that could decrease performance itself) or put instrumentation in own interceptors placed on the very early/late phase of interceptor chain.
Curious how your performance results correlates with these ones: http://ashakirin.blogspot.de/2012/09/cxf-performance-on-sun-solaris-platform.html
Regards,
Andrei.
> -----Original Message-----
> From: Warren, Jared S [mailto:jswarren@jcp.com]
> Sent: Donnerstag, 16. Januar 2014 17:29
> To: users@cxf.apache.org
> Subject: CXF Client performance troubleshooting
>
> Hey, all -
>
> I'm facing an interesting challenge.
>
> I'm using CXF as a web services client, calling a server hosted on a different
> platform.
>
> I've instrumented my code around the call to the CXF port and observe that
> 10 - 20% of the requests perform very poorly (a "good" response time is 100 -
> 200ms, a "bad" one is 4000 - 5000ms, and there are very few requests that
> fall between "good" and "bad").
>
> The server-side log web access log shows that all requests response times
> were "good" (100 - 200ms). For those requests that the server and client
> both think were "good", there is very little latency between the client
> measurements and the server measurements (only a few ms additional time
> in the CXF client than the server thinks the request took). Bandwidth on the
> network segments between the two is nowhere close to saturated. Besides
> that, I don't think network latency would inject SECONDS (perhaps
> milliseconds).
>
> This only occurs under very heavy load, so packet sniffing to figure out which
> side of the wire the problem is on would be extremely arduous (plus, I don't
> have easy access to sniff the relevant network segments).
>
> So...here's my question....
>
> Is there any performance logging I can enable in CXF so that I can see "inside"
> CXF and try to push my visibility "closer to the wire"?
>
>
>
>
>
>
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged material.
> If the reader of this message is not the intended recipient, you are hereby
> notified that your access is unauthorized, and any review, dissemination,
> distribution or copying of this message including any attachments is strictly
> prohibited. If you are not the intended recipient, please contact the sender
> and delete the material from any computer.
Re: CXF Client performance troubleshooting
Posted by rs...@certifydatasystems.com.
Do you use host names or ip addresses?I recall this one time when we were using up addresses on a cloud providers systems and it was still going to a DNS server. Is the time wasted before requests get processed or after? Do you use apache or nginx or something similar to front the server?
---Sent from Boxer | http://getboxer.com
Hey, all -
I'm facing an interesting challenge.
I'm using CXF as a web services client, calling a server hosted on a different platform.
I've instrumented my code around the call to the CXF port and observe that 10 - 20% of the requests perform very poorly (a "good" response time is 100 - 200ms, a "bad" one is 4000 - 5000ms, and there are very few requests that fall between "good" and "bad").
The server-side log web access log shows that all requests response times were "good" (100 - 200ms). For those requests that the server and client both think were "good", there is very little latency between the client measurements and the server measurements (only a few ms additional time in the CXF client than the server thinks the request took). Bandwidth on the network segments between the two is nowhere close to saturated. Besides that, I don't think network latency would inject SECONDS (perhaps milliseconds).
This only occurs under very heavy load, so packet sniffing to figure out which side of the wire the problem is on would be extremely arduous (plus, I don't have easy access to sniff the relevant network segments).
So...here's my question....
Is there any performance logging I can enable in CXF so that I can see "inside" CXF and try to push my visibility "closer to the wire"?
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.